:W&Bá¾÷ÒÿM e44iwgzpª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0LN…Æü‚" 5ou6qxl6ss91 e44iwgzpŠØ e44iwgzpjuju11-uc-san-diegoSPAtop"› _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.02705552540650492 epochs‚10  hidden_dim‚128 ( learning_rate‚0.00018428437211185608  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_vanilla_e44iwgzpb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ £™æÔ°ˆÊ¯š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64QÄ´Ç7 e44iwgzp&  _wandb.runtime‚0  _runtime‚0Î[gzjÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ¢™æÔ°Ï€¤*-of*=/data/spatop/options_files/sweep_tttt_jj/config_e44iwgzp.json*-l* /scratch/logs*-n*jjtttt_vanilla_e44iwgzp2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº 2chgqw1qbnzomw4pcls8mjzqaatj4ige⽑ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€À¶°Ñ"ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º 2chgqw1qbnzomw4pcls8mjzqaatj4igeµª½Ç2  requirements.txt§ Ù×W e44iwgzpZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64ß[²(Y e44iwgzpZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ÛgHô*  e44iwgzpb trainer/global_step2àÉÁy\  e44iwgzpZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64͇Kä/  e44iwgzpb*"trainer/global_step2¥› K  e44iwgzpj: ¤™æÔˆ¦­ù*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ÁÓŽh“  e44iwgzpj  ¤™æÔØݦæî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 38.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ í&À8\ e44iwgzpjÊ ¤™æÔ¨üžç·Trainable params: 40.9 M Non-trainable params: 86 Total params: 40.9 M Total estimated model params size (MB): 163.524 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 PDpƒq e44iwgzpj` ¤™æÔˆ›´çPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ¤Å{ e44iwgzpjj ¤™æÔ°ºçZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). T‚5y e44iwgzpjh ¤™æÔÐß¿çXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). Bd· € e44iwgzpjo ¤™æÔðïÄç_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ÷žäz e44iwgzpji ¤™æÔàÊçYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). _À·0t e44iwgzpjc ¤™æÔ¸·ÏçSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ^î•:q e44iwgzpj` ¤™æÔð¿ÔçPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update).  µOt e44iwgzpjc ¤™æÔ°ïÙçSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ùâ¯Ln e44iwgzpj] ¤™æÔ˜èÞçMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). T`yëm e44iwgzpj\ ¤™æÔ²ãçLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ÜÆPb e44iwgzp*Ð  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"ûǵ\ë e44iwgzpjÙ ¤™æÔðÑ¿ìÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ḬN e44iwgzpj¼ ¤™æÔ€üËì«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ià“=ë e44iwgzpjÙ §™æÔˆ²§ÐÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. (1&uP e44iwgzpj¾ §™æԀطЭ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. î#‡ø‚(:† ³™æÔ°¿Ö gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚33.0 gpu.0.powerWatts‚156.12' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RI ‚(:€ ³™æÔý¿ŠÝ% disk.nvme0c0n1.out‚98.35986328125) proc.memory.availableMB‚ 1504.67578125 network.recv‚42677 disk.dm-1.out‚ 0.58203125 disk.nvme0n1.in‚ 153.93359375# disk.nvme0n1.out‚98.35986328125 disk.drbd1777.in‚0 disk.sda3.in‚0 disk.rbd2.out‚ 0.16455078125 disk.dm-0.in‚ 153.93359375 disk.sda3.out‚ 0.58203125$ memory_percent‚90.81618785858154 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.38193432247544% disk./.usageGB‚1108.3445739746094 disk.rbd2.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2780.95703125 cpu‚25.440573490019403 network.sent‚33586 disk.dm-0.out‚98.35986328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 153.93359375) proc.memory.percent‚16.97361469268799 disk.rbd3.in‚0 disk.sda.out‚ 0.58203125 disk.rbd0.out‚0I¹a ‚(:“ ™æÔ†¶×Õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.618199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aSýÎ!‚(:ù ™æÔÙ disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚98.65283203125 disk.dm-1.in‚0 network.sent‚51774% disk./.usageGB‚1108.3447303771973 disk.rbd1.out‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 0.16455078125 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.93359375" proc.memory.rssMB‚ 2780.9765625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.nvme0n1.in‚ 153.93359375 network.recv‚61466 disk.sda3.out‚1.078125" disk.nvme0c0n1.in‚ 153.93359375 disk.dm-0.out‚98.65283203125$ memory_percent‚90.83449840545654 cpu‚25.334551430714487) disk./.usagePercent‚74.38194481878435 disk.rbd1.in‚0 disk.sda.out‚1.078125% disk.nvme0c0n1.out‚98.65283203125 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.out‚1.078125) proc.memory.availableMB‚ 1501.67578125) proc.memory.percent‚16.97373390197754—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JP²”Š”-$) _wandbruntime‚38  _runtime‚38DG ¿%º trainer/global_step‚49 $ _timestamp‚1.788447947279507e+09 _runtime‚ 38.833986674 _step‚0 & lr-AdamW/pg1‚7.468928232821297e-06 & lr-AdamW/pg2‚7.468928232821297e-06›©$¶Á&¼ & lr-AdamW/pg1‚7.468928232821297e-06 & lr-AdamW/pg2‚7.468928232821297e-06 trainer/global_step‚49 $ _timestamp‚1.788447947279507e+09 _runtime‚ 38.833986674 _step‚0Z…¥Ëh'bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP€ëÞ·x(bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP£x+âg)bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPf!õl*bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP˜ÁŠÃi+be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP='”áo,bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP((µiv-br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¤²x•m.bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPû67DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP“ßÓn1bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPoí2™l2bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÌ¢‡t3bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP°NÞæ_4b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP ¯áj5bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP Ü}}s6bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPRRìÐ_7b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPMºmGi8be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP²ÿì7x9bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPú8ah:bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPDzf;bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP‚);fbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPeH’o?bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPb“|Vn@bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPes ÜpAbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP¶ F‚sBbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP{pŽ&rCbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÄ a¸`Db\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP=²ikEbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPpSï`Fb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP¾K¢hGbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÊž²¸iHbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPà.ëjIbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP[VVtgJbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPAI­ÝrKbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP×¾øM]LbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP£¶ñhMbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPô®©cNb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP¤¸†`bZb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPú 4m[bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP?Vwól\bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPA¨Õâg]bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPž¥t?m^bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP””¿q_bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPMúÍj`bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP†šúCnabj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP ƒlcbb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPOr©qobm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPZ¦*ddpb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP %nqbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPRª—irbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPB7.ñfsbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPJ¾ï‹stbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPuÛ@(gubc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPúÿ è^vbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPW0â]kwbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPÝžÍfxbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPÎîG&gybc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP+ì·tzbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPjÀWf{bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPÜbfl|bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPÞeÃj}bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP„m~bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP‰M"kjbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP6µY+o€bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP|¦²hjbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP‚…4m‚bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP!§Vjƒbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÝ·Îg„bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÖ¢3xo…bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP“b`ôr†bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP19dch‡bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPI|Frˆbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPgﶺq‰bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP-´VThŠbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPé.Öi‹bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPPË+ºjŒbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP D›‘fba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP­Þµ©oŽbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPûT¯pbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPH¥Ÿkbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP‚IJ!f‘ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÅ×zçc’b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÀRÄMh“bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPä· j”be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP?„Þ»t•bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP°²Šq–bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPˆ\tÇq—bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPL Æøm˜bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPjBôÃq™bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPʵ¡mšbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP`po›bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPÅfpœbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPX¸îIsbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP}³§³bžb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPg2P®lŸbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPCÓYHl bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPKü(k¡bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP~ìÜ .¢) _wandbruntime‚38  _runtime‚38鮲à£üN L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 43352637440 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚835 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884479472802656e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚645130 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2875955288576 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚231030 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 40583250658 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚939738 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 41390860288 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2834546425344 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2802144636642 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 trainer/global_step‚49 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2760906955776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396+®Ä†'¤€O ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚231480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2802144636642 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚921198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 trainer/global_step‚49 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2760906955776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 43352637440 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚708396 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚835 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 % _timestamp‚1.7884479472802656e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚645130 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2875955288576 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚231030 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 40583250658 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚939738 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 41390860288 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 40735264768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2834546425344 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2760889916416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚708398 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 41390860288 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2834546425344 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 181444096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¡îPb¥b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP…¬»)_¦bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP©„5v]§bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPóm.p¨bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPQÿÖdf©ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPŸéšUoªbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP„œ±gm«bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP±Ÿ¾h¬bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÛáÞLh­bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPVž%q®bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPG\rÑm¯bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPÁ<éÑw°br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPúY?ei±bd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP:}$Aw²br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPÑyñÝg³bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP0õ½Ïo´bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP‰ü/mµbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPcØ™Œf¶ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPú1Kek·bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPZ(eœo¸bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPo¿Äh¹bc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPàr^hºbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP_çþp»bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPv§s¼bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP_©hm½bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPB6­Mr¾bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP¿Èh÷e¿b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPäTðoÀbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPÈm•dÁb_ gÌbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP`Ø©˜qÍbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPàÓ÷olÎbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP/„ÙjÏbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP] ¡iÐbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP5•¨pÑbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPx’•†iÒbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÐdŽiÓbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP¥«@2cÔb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP ¿¸_ÕbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP±žkrÖbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP!ÏEs×bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP÷oÐrØbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPpPS#gÙbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÔŽzmÚbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPkÂ…fÛba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPà…fÜba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPl·­ÞfÝba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP<*ÞqÞbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPáÊ5hßbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPB³´‡kàbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPwšUeáb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPBhªnâbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP”×iãbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPruù­uäbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÈ|¡Šdåb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP¡¾agëbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP¤í.;dìb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÁߧpõbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP?yë>köbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP׉T,n÷bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPh_3.aøb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPF5÷ mùbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP&'èpúbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP¥æÝÉeûb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPÊë=lübg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPhëªînýbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP›s t^þbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP¿19cdÿb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP#ƒí^‚bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPÆæákƒbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP&û "a„b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPK5ˆ!r…bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÌI(i†bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPA Ózn‡bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP-?¡Úbˆb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPЀ¦@l‰bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPVü×eŠb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP:0o;m‹bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPŸ&F®nŒbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP1çôãhbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPw>ñ.hŽbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPÈíâÇhbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP¦”Š`b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP¦ðÓ(g‘bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP'Ûíj’be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPXíxm“bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPj?Ðm”bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP†’„q•bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP3<]–bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÿCóÚj—be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP•ðWër˜bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP–õîb™b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPSO¢!VšbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP펋 l›bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPþnúrœbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP†Ya÷lbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP…nÃÔežb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPšÙecjŸbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPvt¬w. ) _wandbruntime‚39  _runtime‚39t˜¦äz¡‡M >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚236085 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 44373842432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 % _timestamp‚1.7884479479425995e+09 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 44205270528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚49 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 39.497637271 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 41399685622 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976(GY‘&¢‹M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚422270 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3209024262656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚49 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 39.497637271 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 332015104 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 41399685622 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚939652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚958218 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2890917467136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚236085 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 44373842432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2933127768576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 866123776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 % _timestamp‚1.7884479479425995e+09 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 44205270528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 F loss/SRqqt1/assignment_loss"trainer/global_step2:JPæï—%B§b= loss/SRqqt2/detection_loss"trainer/global_step2:JP§LøO@¨b; loss/FBt4/detection_loss"trainer/global_step2:JPýÍ!-©b( epoch"trainer/global_step2:JPÔwÝùCªb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPo‚pC«b> loss/SRqqt4/assignment_loss"trainer/global_step2:JP…‚V@¬b; loss/FRt2/detection_loss"trainer/global_step2:JP¶k'ë7­b2 loss/total_loss"trainer/global_step2:JP¡L‹gA®b< loss/FRt1/assignment_loss"trainer/global_step2:JP—T­;A¯b< loss/FRt3/assignment_loss"trainer/global_step2:JPƒn£@°b; loss/FRt3/detection_loss"trainer/global_step2:JP˜¦“A±b< loss/FBt4/assignment_loss"trainer/global_step2:JPÀz@²b; loss/FRt4/detection_loss"trainer/global_step2:JPxC½B³b= loss/SRqqt4/detection_loss"trainer/global_step2:JP.š†ê@´b; loss/FBt1/detection_loss"trainer/global_step2:JP•¨ô@µb; loss/FBt3/detection_loss"trainer/global_step2:JPqA¶b< loss/FBt1/assignment_loss"trainer/global_step2:JPÀ©A·b< loss/FBt2/assignment_loss"trainer/global_step2:JPv³Ñ”A¸b< loss/FBt3/assignment_loss"trainer/global_step2:JPr]jâC¹b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPðÉÍp@ºb; loss/FBt2/detection_loss"trainer/global_step2:JPH]ðÞA»b< loss/FRt4/assignment_loss"trainer/global_step2:JPbè­ŠB¼b= loss/SRqqt1/detection_loss"trainer/global_step2:JP Û.½) _wandbruntime‚39  _runtime‚39 ¿.~B¾¼ 1loss/SRqqt3/assignment_loss‚5.363769173622131 -loss/FBt2/detection_loss‚8.11256563379651 /loss/FRt4/assignment_loss‚7.461235098216845 0loss/SRqqt1/detection_loss‚4.014871188185432 .loss/FRt1/detection_loss‚2.921531679558343 1loss/SRqqt3/detection_loss‚3.1027418048335957 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.398038676290801 0loss/SRqqt1/assignment_loss‚5.42462415044958 0loss/SRqqt2/detection_loss‚3.322305546087377 .loss/FBt4/detection_loss‚7.858050189234993 epoch‚0 1loss/SRqqt2/assignment_loss‚5.467635463265812 2loss/SRqqt4/assignment_loss‚5.5274257144412475 .loss/FRt2/detection_loss‚5.215561434175029 %loss/total_loss‚120.8159544945714 _step‚3 /loss/FRt1/assignment_loss‚7.200882360852998 /loss/FRt3/assignment_loss‚7.195173338055611 .loss/FRt3/detection_loss‚4.989534252861631 % _timestamp‚1.7884479479443195e+09 0loss/FBt4/assignment_loss‚2.7560298442840576 /loss/FRt4/detection_loss‚3.5080042967329854 1loss/SRqqt4/detection_loss‚2.0143347992687612 .loss/FBt1/detection_loss‚5.429416340092818 .loss/FBt3/detection_loss‚8.278396512974393 _runtime‚ 39.499708861 /loss/FBt1/assignment_loss‚2.725225559870402 .loss/FBt2/assignment_loss‚2.80403893334525 .loss/FBt3/assignment_loss‚2.724562504074798׊cF¿À /loss/FRt4/assignment_loss‚7.461235098216845 0loss/SRqqt1/detection_loss‚4.014871188185432 .loss/FRt1/detection_loss‚2.921531679558343 1loss/SRqqt3/detection_loss‚3.1027418048335957 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.398038676290801 0loss/SRqqt1/assignment_loss‚5.42462415044958 0loss/SRqqt2/detection_loss‚3.322305546087377 .loss/FBt4/detection_loss‚7.858050189234993 epoch‚0 1loss/SRqqt2/assignment_loss‚5.467635463265812 2loss/SRqqt4/assignment_loss‚5.5274257144412475 .loss/FRt2/detection_loss‚5.215561434175029 %loss/total_loss‚120.8159544945714 _step‚3 /loss/FRt1/assignment_loss‚7.200882360852998 /loss/FRt3/assignment_loss‚7.195173338055611 .loss/FRt3/detection_loss‚4.989534252861631 % _timestamp‚1.7884479479443195e+09 0loss/FBt4/assignment_loss‚2.7560298442840576 /loss/FRt4/detection_loss‚3.5080042967329854 1loss/SRqqt4/detection_loss‚2.0143347992687612 .loss/FBt1/detection_loss‚5.429416340092818 .loss/FBt3/detection_loss‚8.278396512974393 _runtime‚ 39.499708861 /loss/FBt1/assignment_loss‚2.725225559870402 .loss/FBt2/assignment_loss‚2.80403893334525 .loss/FBt3/assignment_loss‚2.72456250407479 1loss/SRqqt3/assignment_loss‚5.363769173622131 -loss/FBt2/detection_loss‚8.11256563379651]柇À‚(:’ Ñ™æÔãçÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’cO Á‚(: Ñ™æÔ¡·†Ù) disk./.usagePercent‚74.38194609882201 disk.dm-0.in‚ 153.93359375! disk.dm-0.out‚196.22509765625 disk.dm-1.out‚ 1.53515625 disk.sda.out‚ 1.53515625 disk.rbd2.in‚0 proc.cpu.threads‚17$ memory_percent‚90.84696769714355" proc.memory.rssMB‚ 2781.0078125 disk.nvme0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1.53515625 disk.rbd0.out‚0 disk.sda.in‚0& disk.nvme0c0n1.out‚196.22509765625) proc.memory.percent‚16.97392463684082% disk./.usageGB‚1108.3447494506836 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.sent‚64769$ disk.nvme0n1.out‚196.22509765625 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 1.87939453125( proc.memory.availableMB‚ 1499.6328125 network.recv‚79752 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 153.93359375 cpu‚25.274219214361846Á‡Î“‚(:ˆ à™æÔ‰¢×Õ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.893' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"§©Ã‚(:û à™æÔÍÕØ% disk.nvme0c0n1.out‚196.8369140625 cpu‚25.235188557243895) disk./.usagePercent‚74.38194609882201 disk.rbd1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 2.02734375# disk.nvme0n1.out‚196.8369140625! proc.memory.rssMB‚ 2781.015625 proc.cpu.threads‚17% disk./.usageGB‚1108.3447494506836 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 2.02734375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 153.93359375$ memory_percent‚90.85311889648438 network.sent‚129127 disk.sda2.in‚0 disk.dm-1.out‚ 2.02734375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚196.8369140625 disk.rbd2.out‚ 1.87939453125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 network.recv‚111702 disk.drbd1777.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 153.93359375$ proc.memory.availableMB‚1498.625) proc.memory.percent‚16.97397232055664 disk.dm-0.in‚ 153.93359375 disk.sda1.in‚0‘oÒ.Ä) _wandbruntime‚74  _runtime‚74Ñ8^­Âż ' lr-AdamW/pg1‚1.5090283572434864e-05 ' lr-AdamW/pg2‚1.5090283572434864e-05 trainer/global_step‚99 $ _timestamp‚1.788447982694773e+09 _runtime‚ 74.249206588 _step‚4Ýç©´ÆÆÀ ' lr-AdamW/pg1‚1.5090283572434864e-05 ' lr-AdamW/pg2‚1.5090283572434864e-05 trainer/global_step‚99 $ _timestamp‚1.788447982694773e+09 _runtime‚ 74.249206588 _step‚4?.Ç) _wandbruntime‚74  _runtime‚74-ª'ȇO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚822502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5587515034482 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 82376666112 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6272075429888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 86346352128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚843 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚5 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1409198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 trainer/global_step‚99 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 81721070592 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1865790 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1865790 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5506113483776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 % _timestamp‚1.7884479826956673e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 86194544128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1847250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 trainer/global_step‚99 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5733919438336 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 81721070592 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1865790 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1865790 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5652216370688 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚459659 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 183540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5506113483776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 % _timestamp‚1.7884479826956673e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1409198 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 86194544128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5652198367744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5734593036800 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 82540498432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 83196107264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 74.930459069 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5643235600006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 82889455446 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚830668 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5643235600006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6334403763200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 82889455446 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 87047887872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚830668 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 82540498432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 183540224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 83196107264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 74.930459069z¢¿.Í) _wandbruntime‚74  _runtime‚74É6&yDξ /loss/FRt1/detection_loss‚2.6658551561335724 /loss/FBt3/detection_loss‚7.1247836315631865 0loss/FBt3/assignment_loss‚2.5961388874053957 /loss/FRt2/assignment_loss‚7.343124414101625 /loss/FRt3/assignment_loss‚7.299426033383324 0loss/FBt2/assignment_loss‚2.6976347390343163 _runtime‚ 74.93182463 _step‚7 1loss/SRqqt3/assignment_loss‚5.419926794563851 1loss/SRqqt1/assignment_loss‚5.327354669570923 0loss/SRqqt4/assignment_loss‚5.50423871146308 0loss/SRqqt1/detection_loss‚3.384807429431627 0loss/SRqqt4/detection_loss‚2.494734438195352 epoch‚0 .loss/FRt2/detection_loss‚4.334194948275884 /loss/SRqqt3/detection_loss‚4.21106778121576 /loss/SRqqt2/detection_loss‚4.24837706074482 % _timestamp‚1.7884479833775373e+09 0loss/FBt4/assignment_loss‚2.6197531784282013 /loss/FRt4/assignment_loss‚7.483858108520508 /loss/FBt1/assignment_loss‚2.606770520116769 .loss/FRt4/detection_loss‚3.348995943762818 .loss/FRt3/detection_loss‚7.218563142277069 .loss/FBt2/detection_loss‚9.905192480367774 /loss/FBt4/detection_loss‚10.318664554287405 &loss/total_loss‚123.49315152005043 trainer/global_step‚99 1loss/SRqqt2/assignment_loss‚5.424244473620159 .loss/FBt1/detection_loss‚2.781193234355134 /loss/FRt1/assignment_loss‚7.134251189231873j^!LHÏ /loss/FRt1/detection_loss‚2.6658551561335724 /loss/FBt3/detection_loss‚7.1247836315631865 0loss/FBt3/assignment_loss‚2.5961388874053957 /loss/FRt2/assignment_loss‚7.343124414101625 /loss/FRt3/assignment_loss‚7.299426033383324 0loss/FBt2/assignment_loss‚2.6976347390343163 _runtime‚ 74.93182463 _step‚7 1loss/SRqqt3/assignment_loss‚5.419926794563851 1loss/SRqqt1/assignment_loss‚5.327354669570923 0loss/SRqqt4/assignment_loss‚5.50423871146308 0loss/SRqqt1/detection_loss‚3.384807429431627 0loss/SRqqt4/detection_loss‚2.494734438195352 epoch‚0 .loss/FRt2/detection_loss‚4.334194948275884 /loss/SRqqt3/detection_loss‚4.21106778121576 /loss/SRqqt2/detection_loss‚4.24837706074482 % _timestamp‚1.7884479833775373e+09 0loss/FBt4/assignment_loss‚2.6197531784282013 /loss/FRt4/assignment_loss‚7.483858108520508 /loss/FBt1/assignment_loss‚2.606770520116769 .loss/FRt4/detection_loss‚3.348995943762818 .loss/FRt3/detection_loss‚7.218563142277069 .loss/FBt2/detection_loss‚9.905192480367774 /loss/FBt4/detection_loss‚10.318664554287405 &loss/total_loss‚123.49315152005043 trainer/global_step‚99 1loss/SRqqt2/assignment_loss‚5.424244473620159 .loss/FBt1/detection_loss‚2.781193234355134 /loss/FRt1/assignment_loss‚7.134251189231873‰@4ܒЂ(:‡ ï™æÔʾüÕ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚33.0 gpu.0.powerWatts‚105.765' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.153 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›”ØÑ‚(:ö ï™æÔÀäØ disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.95751953125$ disk.nvme0c0n1.out‚ 294.708984375 disk.sda2.out‚0) proc.memory.percent‚16.97397232055664 network.recv¡¦"¼)‚116274 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2781.015625" disk.nvme0n1.out‚ 294.708984375( proc.memory.availableMB‚ 1498.7734375 cpu‚25.205045093361722 disk.rbd3.out‚0 disk.sda.out‚2.484375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 153.93359375) disk./.usagePercent‚74.38195147498024 disk.rbd0.in‚0 disk.nvme0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.dm-0.out‚ 294.708984375 disk.sda2.in‚0 disk.sda3.out‚2.484375 disk.dm-0.in‚ 153.93359375 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda.in‚0 network.sent‚135565% disk./.usageGB‚1108.3448295593262 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚2.484375$ memory_percent‚90.85221290588379 proc.cpu.threads‚17Æ7¹“Ò‚(:ˆ þ™æÔ¸íÝÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸­Ó‚(:û þ™æÔá·‘Ù disk.rbd1.in‚0 disk.sda.out‚3.359375 disk.sda3.in‚0 cpu‚25.18261677994132% disk./.usageGB‚1108.3448104858398 disk.nvme0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.dm-0.out‚295.1298828125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚3.359375 disk.rbd3.out‚0) disk./.usagePercent‚74.38195019494258# disk.nvme0n1.out‚295.1298828125 disk.rbd3.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1498.45703125 proc.cpu.threads‚17 network.recv‚122620" disk.nvme0c0n1.in‚ 153.93359375 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 153.93359375 disk.sda3.out‚3.359375 disk.rbd0.out‚0 disk.rbd2.out‚ 1.95751953125 disk.rbd1.out‚0 disk.sda1.in‚0$ memory_percent‚90.85414409637451) proc.memory.percent‚16.97399616241455 network.sent‚185495 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2781.01953125% disk.nvme0c0n1.out‚295.1298828125ÜTïÌžÔ‚(:“ šæÔµüóÕ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.893000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëíâ± Õ‚(:‚ šæÔÆÅÝ× network.recv‚127228 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 2.1064453125& disk.nvme0c0n1.out‚392.85791015625 disk.rbd2.in‚0) disk./.usagePercent‚74.38195019494258 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2781.01953125) proc.memory.percent‚16.97399616241455 disk.dm-1.out‚ 3.55859375$ disk.nvme0n1.out‚392.85791015625 disk.sda.out‚ 3.55859375 disk.sda1.in‚0# memory_percent‚90.8541202545166 proc.cpu.threads‚17% disk./.usageGB‚1108.3448104858398 disk.sda2.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 1498.4609375 disk.dm-0.in‚ 153.93359375 disk.rbd1.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.nvme0n1.in‚ 153.93359375 cpu‚25.166035377983867 network.sent‚191543 disk.drbd1777.in‚0! disk.dm-0.out‚392.85791015625 disk.sda3.in‚0 disk.sda3.out‚ 3.55859375 !É0Ö+ _wandbruntime‚110  _runtime‚110L‚2Ä×¾ ' lr-AdamW/pg1‚2.2711638912048433e-05 ' lr-AdamW/pg2‚2.2711638912048433e-05 trainer/global_step‚149 % _timestamp‚1.7884480184674602e+09 _runtime‚ 110.02181779 _step‚8¢¼2cÈØ ' lr-AdamW/pg1‚2.2711638912048433e-05 ' lr-AdamW/pg2‚2.2711638912048433e-05 trainer/global_step‚149 % _timestamp‚1.7884480184674602e+09 _runtime‚ 110.02181779 _step‚8Útñ0Ù+ _wandbruntime‚110  _runtime‚110ïRŽI™'Ú“O L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8592557186560 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8469868313088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8251302972416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 110.022734115 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2109998 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 123362471936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884480184681637e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1230900 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 122253929474 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2791842 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 trainer/global_step‚149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1230902 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9397364873728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8469850310144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8374228371458 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8251320011776 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8469850310144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8374228371458 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8251320011776 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8469868313088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8251302972416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 110.022734115 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2109998 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 123362471936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 jTùæ0Ü+ _wandbruntime‚110  _runtime‚110b¨kæÝœM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8526203348992 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9460063508992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚692342 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8526221351936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 $ _timestamp‚1.788448019127605e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2124014 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4934336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 123070364438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all:Ë8¼.allocated‚ 8650403265024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8429948937190 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 123070364438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8429948937190 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8526203348992 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9460063508992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1931412 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚692342 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8526221351936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 $ _timestamp‚1.788448019127605e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ÅIr]0ß+ _wandbruntime‚110  _runtime‚110GIxMàÇ 0loss/FBt1/assignment_loss‚1.7913936493826694 .loss/FRt3/detection_loss‚5.069958897829056 .loss/FRt4/detection_loss‚2.695649178404557 /loss/FBt1/detection_loss‚2.1039744168768073 _step‚11 1loss/SRqqt2/assignment_loss‚5.464171131451924 /loss/FBt3/detection_loss‚11.433582428842783 /loss/FBt4/detection_loss‚13.557656219372383 2loss/SRqqt3/assignment_loss‚5.4472528775533045 0loss/FBt2/assignment_loss‚1.8901944041252137 epoch‚0 % _timestamp‚1.7884480191293504e+09 /loss/FRt1/detection_loss‚2.8665915917943825 .loss/FRt2/detection_loss‚6.333499490110962 /loss/SRqqt1/detection_loss‚2.46763669435055 0loss/FBt3/assignment_loss‚1.8755753934383392 0loss/SRqqt3/detection_loss‚5.690806356072426 1loss/SRqqt4/detection_loss‚2.0109056760556996 _runtime‚ 110.683657777 &loss/total_loss‚134.86038383621167 /loss/FRt2/assignment_loss‚7.367978096008301 /loss/FRt4/assignment_loss‚7.494061771192048 0loss/FBt4/assignment_loss‚1.9457951142237737 0loss/SRqqt2/detection_loss‚4.931942452987035 trainer/global_step‚149 -loss/FRt1/assignment_loss‚7.1805044781078 2loss/SRqqt1/assignment_loss‚5.2968565264055805 .loss/FBt2/detection_loss‚17.10104328393936 /loss/FRt3/assignment_loss‚7.333626022338867 1loss/SRqqt4/assignment_loss‚5.509727685347848ŠÎŒQáË .loss/FBt2/detection_loss‚17.10104328393936 /loss/FRt3/assignment_loss‚7.333626022338867 1loss/SRqqt4/assignment_loss‚5.509727685347848 0loss/FBt1/assignment_loss‚1.7913936493826694 .loss/FRt3/detection_loss‚5.069958897829056 .loss/FRt4/detection_loss‚2.695649178404557 /loss/FBt1/detection_loss‚2.1039744168768073 _step‚11 1loss/SRqqt2/assignment_loss‚5.464171131451924 /loss/FBt3/detection_loss‚11.433582428842783 /loss/FBt4/detection_loss‚13.557656219372383 2loss/SRqqt3/assignment_loss‚5.4472528775533045 0loss/FBt2/assignment_loss‚1.8901944041252137 epoch‚0 % _timestamp‚1.7884480191293504e+09 /loss/FRt1/detection_loss‚2.8665915917943825 .loss/FRt2/detection_loss‚6.333499490110962 /loss/SRqqt1/detection_loss‚2.46763669435055 0loss/FBt3/assignment_loss‚1.8755753934383392 0loss/SRqqt3/detection_loss‚5.690806356072426 1loss/SRqqt4/detection_loss‚2.0109056760556996 _runtime‚ 110.683657777 &loss/total_loss‚134.86038383621167 /loss/FRt2/assignment_loss‚7.367978096008301 /loss/FRt4/assignment_loss‚7.494061771192048 0loss/FBt4/assignment_loss‚1.9457951142237737 0loss/SRqqt2/detection_loss‚4.931942452987035 trainer/global_step‚149 -loss/FRt1/assignment_loss‚7.1805044781078 2loss/SRqqt1/assignment_loss‚5.2968565264055805 µ:.´’â‚(:‡ œšæÔ­¶ïÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûwP ã‚(:þ œšæÔÑœ™Ù disk.dm-0.in‚ 153.93359375! disk.dm-0.out‚393.27099609375 disk.sda3.in‚0 disk.sda3.out‚ 4.1953125 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚393.27099609375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0$ memory_percent‚90.85404872894287 network.sent‚198351 network.recv‚132522 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚0 disk.nvme0n1.in‚ 153.93359375 disk.dm-1.in‚0) proc.memory.percent‚16.97406768798828) disk./.usagePercent‚74.38195531509325 disk.sda1.out‚0 disk.rbd2.in‚0% disk./.usageGB‚1108.3448867797852 disk.sda.in‚0 disk.rbd2.out‚ 2.1220703125) proc.memory.availableMB‚ 1498.47265625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.sda.out‚ 4.1953125$ disk.nvme0n1.out‚393.27099609375 disk.dm-1.out‚ 4.1953125 proc.memory.rssMB‚ 2781.03125 cpu‚25.155276699126315ù@,‘ä‚(:† «šæÔž±òÕ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.755' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.151 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08‹‰ å‚(:þ «šæÔŠ÷´Ø disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 4.53515625) disk./.usagePercent‚74.38195531509325 disk.rbd3.in‚0 disk.sda1.out‚0$ memory_percent‚90.85462093353271) proc.memory.percent‚16.97406768798828 proc.cpu.threads‚17% disk./.usageGB‚1108.3448867797852 disk.dm-0.out‚490.9638671875 disk.rbd3.out‚0 disk.sda3.out‚ 4.53515625 disk.sda.in‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚490.9638671875 proc.memory.rssMB‚ 2781.03125 cpu‚25.144142087673437 network.recv‚138075 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.8212890625 disk.dm-0.in‚ 153.93359375 disk.sda2.in‚0 network.sent‚247596 disk.sda.out‚ 4.53515625) proc.memory.availableMB‚ 1498.37890625 disk.rbd1.out‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.nvme0n1.in‚ 153.93359375# disk.nvme0n1.out‚490.9638671875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0Z³R²0æ+ _wandbruntime‚145  _runtime‚1455ÓÄç¾ _runtime‚ 145.83805668 _step‚12 ' lr-AdamW/pg1‚3.0332994251662002e-05 ' lr-AdamW/pg2‚3.0332994251662002e-05 trainer/global_step‚199 $ _timestamp‚1.788448054283666e+09Ù/sðÈè $ _timestamp‚1.788448054283666e+09 _runtime‚ 145.83805668 _step‚12 ' lr-AdamW/pg1‚3.0332994251662002e-05 ' lr-AdamW/pg2‚3.0332994251662002e-05 trainer/global_step‚199 ŠÒfŠ0é+ _wandbruntime‚145  _runtime‚145-‡à}©'ê£O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 145.839580729 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12350802844160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚13 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6510150 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11287520255488 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3699354 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11160270238866 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 $ _timestamp‚1.788448054284704e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3699354 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11451868533248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12522732546048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12522920877056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11160270238866 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 $ _timestamp‚1.788448054284704e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2810796 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12350802844160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚13 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6510150 )%Õ§0ì+ _wandbruntime‚146  _runtime‚146Fè?b´&í®M EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 146.511623175 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 172978330624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3736374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3717808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚14 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11509041013248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11343873294336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 146.511623175 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 172978330624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11051430670336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11509041013248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2824812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3736374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11215319334822 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3717808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚14 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11509041013248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11343873294336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430†8^0ï+ _wandbruntime‚146  _runtime‚146z™8/MðÇ trainer/global_step‚199 1loss/SRqqt2/assignment_loss‚5.089520555554015 1loss/SRqqt4/assignment_loss‚5.032767166962495 0loss/FBt1/assignment_loss‚0.5871475552519162 0loss/FBt3/assignment_loss‚0.6346854211555587 0loss/SRqqt2/detection_loss‚5.419540636467211 /loss/FBt4/detection_loss‚17.373150435090064 1loss/SRqqt1/assignment_loss‚4.912063106657967 /loss/FBt2/assignment_loss‚0.813682245356696 .loss/FRt2/detection_loss‚5.359448214061558 /loss/FBt1/detection_loss‚1.2432754208217678 _step‚15 /loss/SRqqt3/detection_loss‚6.49013970957862 &loss/total_loss‚124.81980313733204 /loss/FRt3/assignment_loss‚7.162854951970718 /loss/FRt3/detection_loss‚6.4073739542680626 _runtime‚ 146.512733385 /loss/FRt2/assignment_loss‚7.293318748474121 1loss/SRqqt3/assignment_loss‚5.082556689227069 /loss/FRt1/detection_loss‚2.1665922349885753 .loss/FRt4/detection_loss‚2.469587737369159 .loss/FBt2/detection_loss‚11.86587943988187 1loss/SRqqt1/detection_loss‚2.3812225003327643 .loss/FBt3/detection_loss‚9.936270164118874 % _timestamp‚1.7884480549583957e+09 /loss/FRt1/assignment_loss‚7.110696758049122 /loss/FRt4/assignment_loss‚7.407931070479136 0loss/FBt4/assignment_loss‚0.8828193962574005 1loss/SRqqt4/detection_loss‚1.6972790249573015 epoch‚0QñË 0loss/FBt1/assignment_loss‚0.5871475552519162 0loss/FBt3/assignment_loss‚0.6346854211555587 0loss/SRqqt2/detection_loss‚5.419540636467211 /loss/FBt4/detection_loss‚17.373150435090064 1loss/SRqqt1/assignment_loss‚4.912063106657967 /loss/FBt2/assignment_loss‚0.813682245356696 .loss/FRt2/detection_loss‚5.359448214061558 /loss/FBt1/detection_loss‚1.2432754208217678 _step‚15 /loss/SRqqt3/detection_loss‚6.49013970957862 &loss/total_loss‚124.81980313733204 /loss/FRt3/assignment_loss‚7.162854951970718 /loss/FRt3/detection_loss‚6.4073739542680626 _runtime‚ 146.512733385 /loss/FRt2/assignment_loss‚7.293318748474121 1loss/SRqqt3/assignment_loss‚5.082556689227069 /loss/FRt1/detection_loss‚2.1665922349885753 .loss/FRt4/detection_loss‚2.469587737369159 .loss/FBt2/detection_loss‚11.86587943988187 1loss/SRqqt1/detection_loss‚2.3812225003327643 .loss/FBt3/detection_loss‚9.936270164118874 % _timestamp‚1.7884480549583957e+09 /loss/FRt1/assignment_loss‚7.110696758049122 /loss/FRt4/assignment_loss‚7.407931070479136 0loss/FBt4/assignment_loss‚0.8828193962574005 1loss/SRqqt4/detection_loss‚1.6972790249573015 epoch‚0 trainer/global_step‚199 1loss/SRqqt2/assignment_loss‚5.089520555554015 1loss/SRqqt4/assignment_loss‚5.032767166962495V­â4‘ò‚(:† ºšæÔЩàÕ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.982 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yâyó‚(:û ºšæÔÁÌœÙ" proc.memory.rssMB‚ 2781.0390625( proc.memory.percent‚16.9741153717041) disk./.usagePercent‚74.38195147498024 disk.sda2.out‚0 disk.rbd1.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 5.3046875 disk.rbd2.in‚0( proc.memory.availableMB‚ 1497.8984375 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚142808% disk.nvme0c0n1.out‚491.3466796875$ memory_percent‚90.85755348205566 cpu‚25.135865100376435 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚491.3466796875 disk.sda2.in‚0 disk.rbd2.out‚ 3.8369140625# disk.nvme0n1.out‚491.3466796875 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 5.3046875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 5.3046875 network.sent‚254261% disk./.usageGB‚1108.3448295593262 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.93359375" disk.nvme0c0n1.in‚ 153.93359375 disk.nvme0n1.in‚ 153.93359375$D’mô‚(:’ ÉšæÔ®ÁÕ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.05' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.010000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö(;% õ‚(:€ ÉšæÔñïÊØ% disk.nvme0c0n1.out‚588.8662109375 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2781.0390625 disk.sda.in‚0 disk.sda3.out‚ 5.62109375 network.sent‚304109 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 5.62109375) proc.memory.availableMB‚ 1498.03515625 cpu‚25.129282883130816 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-0.out‚588.8662109375 disk.rbd1.out‚0 disk.sda.out‚ 5.62109375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 3.98583984375# disk.nvme0n1.out‚588.8662109375 disk.dm-0.in‚ 153.93359375 disk.dm-1.in‚0$ memory_percent‚90.85671901702881( proc.memory.percent‚16.9741153717041 network.recv‚149304) disk./.usagePercent‚74.38195147498024" disk.nvme0c0n1.in‚ 153.93359375% disk./.usageGB‚1108.3448295593262 disk.rbd3.in‚0 disk.rbd3.out‚0ON~“ö‚(:ˆ ØšæÔù·‘× gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.928' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&a°÷‚(:û ØšæÔÀª™Ø disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 6.0703125 disk.rbd2.in‚0 disk.rbd2.out‚ 4.00146484375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 153.93359375 disk.dm-1.in‚0 disk.dm-0.in‚ 153.93359375! disk.dm-0.out‚589.20361328125 disk.dm-1.out‚ 6.0703125( proc.memory.percent‚16.9741153717041$ disk./.usageGB‚1108.344970703125 disk.sda3.in‚0& disk.nvme0c0n1.out‚589.20361328125 network.sent‚310810' disk./.usagePercent‚74.381960947259 disk.rbd1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.123598759372744 network.recv‚154288 disk.rbd3.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚589.20361328125 disk.sda2.in‚0$ memory_percent‚90.85683822631836' proc.memory.availableMB‚ 1498.015625" proc.memory.rssMB‚ 2781.0390625 disk.sda3.out‚ 6.0703125æ˼S0ø+ _wandbruntime‚181  _runtime‚181ëtüÄù¾ & lr-AdamW/pg1‚3.795434959127557e-05 & lr-AdamW/pg2‚3.795434959127557e-05 trainer/global_step‚249 % _timestamp‚1.7884480898399637e+09 _runtime‚ 181.394443092 _step‚16ƒ¥¥"Jú & lr-AdamW/pg1‚3.795434959127557e-05 & lr-AdamW/pg2‚3.7954349õN0±~59127557e-05 trainer/global_step‚249 % _timestamp‚1.7884480898399637e+09 _runtime‚ 181.394443092 _step‚16IÖ7è0û+ _wandbruntime‚181  _runtime‚181î(;œ«'ü¥O E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15648345364480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14105172197888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205334083584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14309832683008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3194523 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1146821 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13945640636706 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13741716028416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 205334083584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788448089840872e+09$ª›Ö¯'ý©O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203924608290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4625406 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14105172197888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8137002 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 205334083584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14309832683008 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3194523 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1146821 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13945640636706 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13741716028416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 205334083584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3511598 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788448089840872e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1146382 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15648345364480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15710852578304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14161507233792 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 215693541376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚ 182.055193056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14367678761472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3525614 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13796620158976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14367678761472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚879 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14161525236736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14001361202230 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2055868 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 trainer/global_step‚249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚18 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14161525236736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15494966524928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844Ò–¯VဴM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14161525236736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15494966524928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4662426 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15710852578304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14161507233792 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 4.DeviÎÛªZÙceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 215693541376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3525612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204741043254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚ 182.055193056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14367678761472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3525614 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13796620158976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14367678761472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚879 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14161525236736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14001361202230 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2055868 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 trainer/global_step‚249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚18 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„e0+ _wandbruntime‚182  _runtime‚182ÓuóG‚Á .loss/FBt1/detection_loss‚1.173847636917165 /loss/FBt2/assignment_loss‚0.446931197160953 -loss/FRt4/detection_loss‚2.70624780295224 /loss/SRqqt3/detection_loss‚4.09337887458685 .loss/FBt4/detection_loss‚7.749903429638255 epoch‚0 /loss/FRt3/assignment_loss‚7.134174108505249 /loss/FRt4/assignment_loss‚7.447206316323116 0loss/SRqqt2/assignment_loss‚3.89551842212677 _runtime‚ 182.056898208 1loss/SRqqt3/assignment_loss‚3.914008791853742 0loss/FBt1/assignment_loss‚0.3414950268343091 -loss/FRt2/detection_loss‚4.43265599776537 0loss/SRqqt2/detection_loss‚4.290178860723972 -loss/FBt3/detection_loss‚12.7693482956716 trainer/global_step‚249 _step‚19 0loss/SRqqt4/detection_loss‚2.412000253196381 2loss/SRqqt4/assignment_loss‚3.9739270347815294 &loss/total_loss‚102.42163526442855 % _timestamp‚1.7884480905025573e+09 .loss/FRt2/assignment_loss‚7.23179064041529 2loss/SRqqt1/assignment_loss‚3.4028760484746985 0loss/FBt3/assignment_loss‚0.5285999331224177 1loss/SRqqt1/detection_loss‚1.7301120514905937 .loss/FBt2/detection_loss‚8.794905293928949 1loss/FBt4/assignment_loss‚0.40337424708360975 .loss/FRt3/detection_loss‚4.024676328668227 /loss/FRt1/assignment_loss‚7.249025655829388 /loss/FRt1/detection_loss‚2.2754530163778774>ÂÒKƒÅ 1loss/SRqqt3/assignment_loss‚3.914008791853742 0loss/FBt1/assignment_loss‚0.3414950268343091 -loss/FRt2/detection_loss‚4.43265599776537 0loss/SRqqt2/detection_loss‚4.290178860723972 -loss/FBt3/detection_loss‚12.7693482956716 trainer/global_step‚249 _step‚19 0loss/SRqqt4/detection_loss‚2.412000253196381 2loss/SRqqt4/assignment_loss‚3.9739270347815294 &loss/total_loss‚102.42163526442855 % _timestamp‚1.7884480905025573e+09 .loss/FRt2/assignment_loss‚7.23179064041529 2loss/SRqqt1/assignment_loss‚3.4028760484746985 0loss/FBt3/assignment_loss‚0.5285999331224177 1loss/SRqqt1/detection_loss‚1.7301120514905937 .loss/FBt2/detection_loss‚8.794905293928949 1loss/FBt4/assignment_loss‚0.40337424708360975 .loss/FRt3/detection_loss‚4.024676328668227 /loss/FRt1/assignment_loss‚7.249025655829388 /loss/FRt1/detection_loss‚2.2754530163778774 .loss/FBt1/detection_loss‚1.173847636917165 /loss/FBt2/assignment_loss‚0.446931197160953 -loss/FRt4/detection_loss‚2.70624780295224 /loss/SRqqt3/detection_loss‚4.09337887458685 .loss/FBt4/detection_loss‚7.749903429638255 epoch‚0 /loss/FRt3/assignment_loss‚7.134174108505249 /loss/FRt4/assignment_loss‚7.447206316323116 0loss/SRqqt2/assignment_loss‚3.89551842212677 _runtime‚ 182.056898208+®‚“„‚(:ˆ çšæÔ ¤ÌÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È2}…‚(:õ çšæÔŠòÿØ disk.sda3.out‚6.4375 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚686.8740234375$ memory_percent‚90.85705280303955 network.sent‚316507 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚74.38195966722134% disk.nvme0c0n1.out‚686.8740234375 disk.dm-1.out‚6.4375 proc.cpu.threads‚17 disk.sda.out‚6.4375 disk.rbd2.out‚ 4.12646484375 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.93359375) proc.memory.availableMB‚ 1497.98046875% disk./.usageGB‚1108.3449516296387 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.nvme0n1.in‚ 153.93359375! proc.memory.rssMB‚ 2781.046875 network.recv‚158595# disk.nvme0n1.out‚686.8740234375* proc.memory.percent‚16.974163055419922 cpu‚25.118406193490316VŽK_’†‚(:‡ öšæÔî—³Õ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÔ> ‡‚(:€ öšæÔÔ’¥Ù disk.dm-0.out‚687.1865234375$ disk./.usageGB‚1108.344955444336 disk.rbd1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 7.12890625 network.sent‚366816 network.recv‚164762 disk.rbd1.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 7.12890625 disk.rbd2.in‚0 disk.dm-0.in‚ 153.93359375) proc.memory.availableMB‚ 1497.79296875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 7.12890625 disk.nvme0n1.in‚ 153.93359375 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2781.046875* proc.memory.percent‚16.974163055419922 proc.cpu.threads‚17) disk./.usagePercent‚74.38195992322886 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚687.1865234375$ memory_percent‚90.85819721221924 disk.rbd2.out‚ 5.71240234375 disk.sda2.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚687.1865234375 cpu‚25.113987101123218?Ëï0ˆ+ _wandbruntime‚217  _runtime‚217vQFĉ¾ _runtime‚ 217.022902567 _step‚20 & lr-AdamW/pg1‚4.557570493088914e-05 & lr-AdamW/pg2‚4.557570493088914e-05 trainer/global_step‚299 % _timestamp‚1.7884481254685564e+09±úÈŠÂ _step‚20 & lr-AdamW/pg1‚4.557570493088914e-05 & lr-AdamW/pg2‚4.557570493088914e-05 trainer/global_step‚299 % _timestamp‚1.7884481254685564e+09 _runtime‚ 217.022902567ÖŸT0‹+ _wandbruntime‚217  _runtime‚217Eδæ«'Œ¥O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18773716016128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 _runtime‚ 217.02404022 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18515852670464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18773521393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚21 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16732353973682 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3833431 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17169144029696 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9782396 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257839405568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17169144029696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16922824140288 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477gÕÞZ<"©O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16732353973682 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3832986 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3833431 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17169144029696 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9782396 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257839405568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17169144029696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16922824140288 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 245664293888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18773716016128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 _runtime‚ 217.02404022 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18515852670464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18773521393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 244759947698 ?, ã.s9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚21 H6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4226412 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 _step‚22 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17226316509696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17226316509696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 _runtime‚ 217.695533319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16979159176192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1381465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4226414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 _runtime‚ 217.695533319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16979159176192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1381465 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4226414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3846181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16541843726336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16979177179136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5569912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4226412 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4226414 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4226412 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 188317696 _step‚22 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17226316509696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17226316509696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9796324å>bZ0‘+ _wandbruntime‚217  _runtime‚217ÑN‘R’Ì /loss/FRt3/assignment_loss‚7.269288756630638 2loss/SRqqt2/assignment_loss‚3.3127684189101396 0loss/FBt1/assignment_loss‚0.4323675655759871 epoch‚0 0loss/FBt4/assignment_loss‚0.3036049067261903 /loss/FRt1/detection_loss‚2.9297535412565425 0loss/SRqqt2/detection_loss‚2.721273366930121 /loss/FBt1/detection_loss‚3.2027151832529657 /loss/FBt2/detection_loss‚16.402126637101173 _runtime‚ 217.697141979 /loss/FRt1/assignment_loss‚7.195184071858724 0loss/FBt2/assignment_loss‚0.5024871217086911 .loss/FRt3/detection_loss‚4.661515928127549 0loss/SRqqt1/detection_loss‚1.723752013937785 .loss/FRt4/detection_loss‚3.118524821299427 1loss/SRqqt4/detection_loss‚1.6627302561745259 % _timestamp‚1.7884481261428204e+09 /loss/FRt2/detection_loss‚3.2616900036147998 /loss/FRt2/assignment_loss‚7.276614048901727 1loss/SRqqt1/assignment_loss‚3.264042802957388 2loss/SRqqt4/assignment_loss‚3.8663620074590046 1loss/FBt3/assignment_loss‚0.12598437199679513 .loss/FBt3/detection_loss‚5.710566552480062 &loss/total_loss‚105.49757279954966 /loss/FRt4/assignment_loss‚7.397310589844325 0loss/SRqqt3/detection_loss‚5.420355518018046 2loss/SRqqt3/assignment_loss‚3.7762088852543987 .loss/FBt4/detection_loss‚9.960345429532667 trainer/global_step‚299 _step‚23Kˆ¥¬V“Ð /loss/FRt3/assignment_loss‚7.269288756630638 2loss/SRqqt2/assignment_loss‚3.3127684189101396 0loss/FBt1/assignment_loss‚0.4323675655759871 epoch‚0 0loss/FBt4/assignment_loss‚0.3036049067261903 /loss/FRt1/detection_loss‚2.9297535412565425 0loss/SRqqt2/detection_loss‚2.721273366930121 /loss/FBt1/detection_loss‚3.2027151832529657 /loss/FBt2/detection_loss‚16.402126637101173 _runtime‚ 217.697141979 /loss/FRt1/assignment_loss‚7.195184071858724 0loss/FBt2/assignment_loss‚0.5024871217086911 .loss/FRt3/detection_loss‚4.661515928127549 0loss/SRqqt1/detection_loss‚1.723752013937785 .loss/FRt4/detection_loss‚3.118524821299427 1loss/SRqqt4/detection_loss‚1.6627302561745259 % _timestamp‚1.7884481261428204e+09 /loss/FRt2/detection_loss‚3.2616900036147998 /loss/FRt2/assignment_loss‚7.276614048901727 1loss/SRqqt1/assignment_loss‚3.264042802957388 2loss/SRqqt4/assignment_loss‚3.8663620074590046 1loss/FBt3/assignment_loss‚0.12598437199679513 .loss/FBt3/detection_loss‚5.710566552480062 &loss/total_loss‚105.49757279954966 /loss/FRt4/assignment_loss‚7.397310589844325 0loss/SRqqt3/detection_loss‚5.420355518018046 2loss/SRqqt3/assignment_loss‚3.7762088852543987 .loss/FBt4/detection_loss‚9.960345429532667 trainer/global_step‚299 _step‚23e' E”‚(:’ …›æÔµ“ÁÖ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚99.706' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.941200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$íæ•‚(:ù …›æÔ›˜¥Ø disk.rbd1.in‚0 disk.rbd3.in‚0 network.recv‚169663 disk.sda3.out‚7.53125" disk.nvme0c0n1.in‚ 153.93359375 disk.nvme0n1.in‚ 153.93359375$ disk.nvme0n1.out‚785.16259765625! disk.dm-0.out‚785.16259765625 disk.rbd3.out‚0 disk.drbd1777.out‚0# memory_percent‚90.8573865890503* proc.memory.percent‚16.974163055419922 cpu‚25.109367039024946 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚785.16259765625 disk.sda2.in‚0) proc.memory.availableMB‚ 1497.92578125 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd2.out‚ 5.85302734375 disk.dm-0.in‚ 153.93359375 disk.dm-1.out‚7.53125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚1108.345199584961 disk.sda.out‚7.53125 disk.rbd0.in‚0 disk.rbd1.out‚0! proc.memory.rssMB‚ 2781.046875 network.sent‚373022) disk./.usagePercent‚74.38197630771106 disk.rbd2.in‚0 disk.drbd1777.in‚0‹¤ ª–‚(:’ ”›æÔØÉ€× gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.665800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îþ® —‚(:‚ ”›æÔˆìÑÙ network.recv‚176097" disk.nvme0c0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 5.87646484375 disk.dm-0.in‚ 153.93359375 disk.sda.in‚0 disk.sda3.out‚ 7.98046875! proc.memory.rssMB‚ 2781.046875& disk.nvme0c0n1.out‚785.48291015625 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚90.85707664489746( proc.memory.availableMB‚ 1497.9765625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda.out‚ 7.98046875! disk.dm-0.out‚785.48291015625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.93359375$ disk.nvme0n1.out‚785.48291015625 disk.sda2.out‚0 disk.dm-1.out‚ 7.98046875* proc.memory.percent‚16.974163055419922 cpu‚25.106692243308714 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.38197630771106$ disk./.usageGB‚1108.345199584961 disk.drbd1777.in‚0 network.sent‚423396B5Pª0˜+ _wandbruntime‚252  _runtime‚252¤ÌsÅ™¿ % _timestamp‚1.7884481610282345e+09 _runtime‚ 252.58260102 _step‚24 ' lr-AdamW/pg1‚5.3197060270502706e-05 ' lr-AdamW/pg2‚5.3197060270502706e-05 trainer/global_step‚349e£Üɚà _step‚24 ' lr-AdamW/pg1‚5.3197060270502706e-05 ' lr-AdamW/pg2‚5.3197060270502706e-05 trainer/global_step‚349 % _timestamp‚1.7884481610282345e+09 _runtime‚ 252.58260102í(Û0›+ _wandbruntime‚252  _runtime‚252÷äHnœªO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19232129084416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6477510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 287305695232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 286650099712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21598413493760 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4913196 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21899039115776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 285595287106 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19232146123776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4472542 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6477510 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21899233738240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚4ÊÄÎßB14 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19740476082688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 300649562112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884481610289652e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 287305695232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0­ƒV»´'®O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884481610289652e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 287305695232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19232129084416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6477510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 287305695232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 286650099712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21598413493760 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4913196 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21899039115776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 285595287106 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19232146123776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4472542 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6477510 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21899233738240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19740476082688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 300649562112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚349 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21660064231424 _runtime‚ 253.254553442 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0oy‚˜ý ¸M ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚349 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4927212 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21660064231424 _runtime‚ 253.254553442 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20084954257920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMo(ÆLÁnitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464|"ˆý0¡+ _wandbruntime‚253  _runtime‚253´ÿmÑO¢É 0loss/FBt3/assignment_loss‚0.2624348760897143 1loss/SRqqt2/detection_loss‚2.8569527169068656 /loss/FBt1/detection_loss‚1.6923394093303106 /loss/FRt4/assignment_loss‚7.434974800456654 2loss/SRqqt4/assignment_loss‚3.7401133464730303 /loss/FRt2/detection_loss‚3.6676427145367083 1loss/SRqqt4/detection_loss‚1.5536646270256218 .loss/FBt3/detection_loss‚4.480950198463492 epoch‚0 .loss/FBt2/detection_loss‚8.034137949347496 2loss/SRqqt2/assignment_loss‚3.2698242840943514 0loss/FBt2/assignment_loss‚0.7056040880270302 -loss/FRt1/detection_loss‚2.76952810802243 /loss/FRt4/detection_loss‚3.7593410604379396 .loss/FRt3/detection_loss‚3.905394965694064 1loss/SRqqt1/detection_loss‚1.4032198215136304 %loss/total_loss‚90.80155084396736 % _timestamp‚1.7884481617018924e+09 2loss/SRqqt1/assignment_loss‚3.1263909570632444 1loss/SRqqt3/assignment_loss‚3.534789589735178 0loss/FBt4/assignment_loss‚0.5283732783884324 _step‚27 /loss/FRt1/assignment_loss‚7.126234808835116 /loss/FRt2/assignment_loss‚7.368103006611699 /loss/FRt3/assignment_loss‚7.011064234234038 .loss/FBt4/detection_loss‚5.542367144905287 _runtime‚ 253.25621961 0loss/FBt1/assignment_loss‚0.6986229299537597 0loss/SRqqt3/detection_loss‚6.329481927821269 trainer/global_step‚349 ëS£Í epoch‚0 .loss/FBt2/detection_loss‚8.034137949347496 2loss/SRqqt2/assignment_loss‚3.2698242840943514 0loss/FBt2/assignment_loss‚0.7056040880270302 -loss/FRt1/detection_loss‚2.76952810802243 /loss/FRt4/detection_loss‚3.7593410604379396 .loss/FRt3/detection_loss‚3.905394965694064 1loss/SRqqt1/detection_loss‚1.4032198215136304 %loss/total_loss‚90.80155084396736 % _timestamp‚1.7884481617018924e+09 2loss/SRqqt1/assignment_loss‚3.1263909570632444 1loss/SRqqt3/assignment_loss‚3.534789589735178 0loss/FBt4/assignment_loss‚0.5283732783884324 _step‚27 /loss/FRt1/assignment_loss‚7.126234808835116 /loss/FRt2/assignment_loss‚7.368103006611699 /loss/FRt3/assignment_loss‚7.011064234234038 .loss/FBt4/detection_loss‚5.542367144905287 _runtime‚ 253.25621961 0loss/FBt1/assignment_loss‚0.6986229299537597 0loss/SRqqt3/detection_loss‚6.329481927821269 trainer/global_step‚349 0loss/FBt3/assignment_loss‚0.2624348760897143 1loss/SRqqt2/detection_loss‚2.8569527169068656 /loss/FBt1/detection_loss‚1.6923394093303106 /loss/FRt4/assignment_loss‚7.434974800456654 2loss/SRqqt4/assignment_loss‚3.7401133464730303 /loss/FRt2/detection_loss‚3.6676427145367083 1loss/SRqqt4/detection_loss‚1.5536646270256218 .loss/FBt3/detection_loss‚4.480950198463492¡š#Ꞥ‚(:“ £›æÔé±ÚÕ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.912799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÂ^ÿ¥‚(:ô £›æÔýöŠØ network.recv‚180583 disk.rbd3.in‚0) proc.memory.availableMB‚ 1497.11328125" disk.nvme0c0n1.in‚ 153.93359375 disk.sda2.out‚0 disk.sda.out‚8.4375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚882.97119140625 disk.sda.in‚0* proc.memory.percent‚16.974163055419922) disk./.usagePercent‚74.38196939550764 disk.dm-0.in‚ 153.93359375 cpu‚25.10394974043062 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda3.in‚0% disk./.usageGB‚1108.3450965881348 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 153.93359375$ disk.nvme0n1.out‚882.97119140625# memory_percent‚90.8623456954956 disk.dm-1.out‚8.4375 disk.rbd3.out‚0 disk.sda3.out‚8.4375& disk.nvme0c0n1.out‚882.97119140625! proc.memory.rssMB‚ 2781.046875 proc.cpu.threads‚17 network.sent‚429175 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 6.025390625ÿ½z“¦‚(:ˆ ²›æÔ퀲Õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³h( §‚(:ÿ ²›æÔ㹃Ù network.recv‚186268 disk.sda3.in‚0 disk.sda3.out‚ 9.10546875! disk.dm-0.out‚883.58251953125 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 9.10546875 disk.sda.out‚ 9.10546875 disk.rbd1.in‚0! proc.memory.rssMB‚ 2781.046875" disk.nvme0c0n1.in‚ 153.93359375 disk.dm-0.in‚ 153.93359375 disk.sda2.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1108.3450965881348 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚90.86294174194336 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.out‚ 6.041015625 disk.drbd1777.out‚0* proc.memory.percent‚16.974163055419922 cpu‚25.10020224883225 network.sent‚479334) disk./.usagePercent‚74.38196939550764 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 153.93359375 disk.rbd3.out‚0 disk.sda.in‚0' proc.memory.availableMB‚ 1497.015625& disk.nvme0c0n1.out‚883.58251953125$ disk.nvme0n1.out‚883.58251953125 disk.rbd1.out‚0™o-Ìž¨‚(:“ Á›æÔí¿Õ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.755000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Q©‚(:ü Á›æÔþâ× network.recv‚190948 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 9.48046875 disk.rbd2.out‚ 6.119140625 disk.drbd1777.out‚0 disk.sda2.out‚0# memory_percent‚90.8595085144043 disk.nvme0n1.in‚ 153.93359375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚981.5712890625 disk.dm-0.out‚981.5712890625" proc.memory.rssMB‚ 2781.0546875 disk.dm-1.out‚ 9.48046875 disk.rbd1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 9.48046875 network.sent‚485306% disk./.usageGB‚1108.3450965881348 disk.sda1.out‚0 disk.rbd2.in‚0% disk.nvme0c0n1.out‚981.5712890625 disk.sda2.in‚0' proc.memory.availableMB‚ 1497.578125) disk./.usagePercent‚74.38196939550764 disk.rbd0.in‚0* proc.memory.percent‚16.974210739135742 cpu‚25.09879245551404 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 153.93359375 disk.dm-0.in‚ 153.93359375=Ró…0ª+ _wandbruntime‚288  _runtime‚288O1ðÅ«¿ ' lr-AdamW/pg1‚6.0818415610116275e-05 ' lr-AdamW/pg2‚6.0818415610116275e-05 trainer/global_step‚399 % _timestamp‚1.7884481968126228e+09 _runtime‚ 288.36704081 _step‚28vG“ˆÉ¬Ã ' lr-AdamW/pg2‚6.0818415610116275e-05 trainer/global_step‚399 % _timestamp‚1.7884481968126228e+09 _runtime‚ 288.36704081 _step‚28 ' lr-AdamW/pg1‚6.0818415610116275e-05Gdí@0­+ _wandbruntime‚288  _runtime‚288ÑLÉÊ°'®ªO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 327085056722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5613998 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22885745927680 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7403562 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884481968133857e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5613998 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 326430626514 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7422102 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚399 _step‚29 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25024954822656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1839288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 288.367969329 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25024753908736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884481968133857e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22886419526144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5613998 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3272902 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 326430626514 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7422102 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 343827471872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚399 _step‚29 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25024954822656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1839288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 288.367969329 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200913920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25024753908736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22032239742976 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 329110942208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1843864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 ?5DeviceStatsMonitor.on_train_batch_end/active.aÏJÑ,ll.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22614463060992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22943592006144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 329110942208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 289.053750194 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 328455333376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 328455333376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440582 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7422016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 % _timestamp‚1.7884481974984725e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5628014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 329110942208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1843864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1844320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22614463060992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22943592006144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 329110942208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 289.053750194 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 328455333376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3281068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 328455333376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7440582 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7422016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 % _timestamp‚1.7884481974984725e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5628014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22032239742976 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28150573232640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25375779967488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387085972480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25375761964544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6314796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605þ?Ž³'¿­O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8329614 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27763487260160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8348154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28150573232640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25375779967488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2071980 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 387085972480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6314798 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25375761964544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6314796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25802229754368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14676880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689468 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25145528671862 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28213093007872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25432115003392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28212902593024 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24777446270976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884482327917054e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27825137997824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 324.346763008 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6328812 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470Ú£8_¾&¸M ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8366634 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 324.346763008 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6328812 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 368736831302 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚25801556142592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3689470 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 370096748032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25802229754368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14676880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689468 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25145528671862 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28213093007872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25432115003392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28212902593024 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 387955010048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27825114057728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24777446270976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884482327917054e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14695448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27825137997824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973"Ýà 0Ã+ _wandbruntime‚324  _runtime‚324AÔÂßPÄÊ 2loss/SRqqt2/assignment_loss‚2.9765537765718277 1loss/FBt3/assignment_loss‚0.43267044522993575 1loss/SRqqt1/detection_loss‚1.7104741279433318 /loss/FRt2/assignment_loss‚7.256707797447841 0loss/SRqqt2/detection_loss‚2.503002347364541 /loss/FRt1/assignment_loss‚7.118522541863578 2loss/SRqqt1/assignment_loss‚2.7630731114025773 .loss/FBt3/detection_loss‚5.653766836722692 _step‚35 0loss/FBt1/assignment_loss‚0.6109676492356119 .loss/FRt4/detection_loss‚4.180226668312743 %loss/total_loss‚94.05382065425158 .loss/FBt1/detection_loss‚2.188205405411801 % _timestamp‚1.7884482327936423e+09 .loss/FRt3/assignment_loss‚7.16986380304609 /loss/FRt4/assignment_loss‚7.371522993654819 1loss/FBt4/assignment_loss‚0.30284491612368986 .loss/FRt1/detection_loss‚2.621366758315292 .loss/FRt3/detection_loss‚5.540695743901389 0loss/SRqqt3/detection_loss‚4.275377371201389 1loss/SRqqt4/detection_loss‚3.2493598284819423 /loss/FBt2/detection_loss‚13.020081507662931 .loss/FBt4/detection_loss‚3.227071581615342 /loss/FRt2/detection_loss‚3.3670860923205814 epoch‚0 2loss/SRqqt3/assignment_loss‚2.6227354736704576 1loss/SRqqt4/assignment_loss‚3.524297368526459 0loss/FBt2/assignment_loss‚0.3673465082247276 trainer/global_step‚449 _runtime‚ 324.347951637Œ„xqöÅÎ 2loss/SRqqt1/assignment_loss‚2.7630731114025773 .loss/FBt3/detection_loss‚5.653766836722692 _step‚35 0loss/FBt1/assignment_loss‚0.6109676492356119 .loss/FRt4/detection_loss‚4.180226668312743 %loss/total_loss‚94.053^’Rõ^82065425158 .loss/FBt1/detection_loss‚2.188205405411801 % _timestamp‚1.7884482327936423e+09 .loss/FRt3/assignment_loss‚7.16986380304609 /loss/FRt4/assignment_loss‚7.371522993654819 1loss/FBt4/assignment_loss‚0.30284491612368986 .loss/FRt1/detection_loss‚2.621366758315292 .loss/FRt3/detection_loss‚5.540695743901389 0loss/SRqqt3/detection_loss‚4.275377371201389 1loss/SRqqt4/detection_loss‚3.2493598284819423 /loss/FBt2/detection_loss‚13.020081507662931 .loss/FBt4/detection_loss‚3.227071581615342 /loss/FRt2/detection_loss‚3.3670860923205814 epoch‚0 2loss/SRqqt3/assignment_loss‚2.6227354736704576 1loss/SRqqt4/assignment_loss‚3.524297368526459 0loss/FBt2/assignment_loss‚0.3673465082247276 trainer/global_step‚449 _runtime‚ 324.347951637 2loss/SRqqt2/assignment_loss‚2.9765537765718277 1loss/FBt3/assignment_loss‚0.43267044522993575 1loss/SRqqt1/detection_loss‚1.7104741279433318 /loss/FRt2/assignment_loss‚7.256707797447841 0loss/SRqqt2/detection_loss‚2.503002347364541 /loss/FRt1/assignment_loss‚7.118522541863578#»0øµžÆ‚(:“ î›æÔõûØÕ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.409' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.081799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ówtÇ‚(:ü î›æÔ ùŠÙ disk.sda1.out‚0 disk.nvme0n1.in‚ 153.93359375 disk.dm-0.in‚ 153.93359375 disk.dm-0.out‚1080.490234375 proc.memory.rssMB‚ 2781.0625 network.recv‚205980) disk./.usagePercent‚74.38199422823845 disk.rbd0.out‚0% disk./.usageGB‚1108.3454666137695# disk.nvme0n1.out‚1080.490234375 disk.sda2.out‚0 disk.sda.in‚0 network.sent‚547892 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 11.62890625* proc.memory.percent‚16.974258422851562" disk.nvme0c0n1.in‚ 153.93359375 disk.drbd1777.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 1497.65625 disk.rbd2.out‚ 7.849609375 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0# memory_percent‚90.8590316772461 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-1.out‚ 11.62890625 disk.sda.out‚ 11.62890625 disk.sda3.in‚0 cpu‚25.091724049070937% disk.nvme0c0n1.out‚1080.490234375 disk.sda2.in‚0 disk.rbd1.out‚0ߟú‘È‚(:† ý›æÔųÕ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚104.16' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.832 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´NÌDÉ‚(:ø ý›æÔÐùóØ disk.rbd1.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2781.0625 network.recv‚212004# disk.nvme0n1.out‚1178.052734375 disk.sda.in‚0 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 disk.dm-1.out‚ 11.8359375 disk.sda.out‚ 11.8359375$ memory_percent‚90.86141586303711 network.sent‚597577 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚1178.052734375 disk.rbd1.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1497.265625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 11.8359375 disk.rbd2.out‚ 7.93603515625! disk.nvme0c0n1.in‚ 153.9765625( disk./.usagePercent‚74.3820049805549 disk.sda3.in‚0* proc.memory.percent‚16.974258422851562 cpu‚25.09073569325128 proc.cpu.threads‚17% disk./.usageGB‚1108.3456268310547 disk.dm-0.out‚1178.052734375Ä…,~0Ê+ _wandbruntime‚359  _runtime‚359¥&ñÖÃ˽ & lr-AdamW/pg2‚7.606112628934342e-05 trainer/global_step‚499 % _timestamp‚1.7884482675796316e+09 _runtime‚ 359.13399164 _step‚36 & lr-AdamW/pg1‚7.606112628934342e-056ÕËÇÌÁ & lr-AdamW/pg1‚7.606112628934342e-05 & lr-AdamW/pg2‚7.606112628934342e-05 trainer/global_step‚499 % _timestamp‚1.7884482675796316e+09 _runtime‚ 359.13399164 _step‚36$¾³2»0Í+ _wandbruntime‚359  _runtime‚359©’ê°'ΪO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚499 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28603695022592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 430126688768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27875849973746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28193431909888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6392618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9255666 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7015598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27467765707776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9274206 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7015598 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28193431909888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2302479 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0cؾ´'Ï®O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28193413906944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28603695022592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30846024143360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 430280593920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 408101305330 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7015596 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9255666 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16289804 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28193413906944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4089702 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2302916 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7015598 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28193431909888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2302479 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚499 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28603695022592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 430126688768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27875849973746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28193431909888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6392618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9255666 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7015598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27467765707776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9274206 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410426945024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2307525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28660193890816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4097868 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚38 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 TFDeviceStatsMonitor+íœd*.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 _runtime‚ 359.795699911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 408917740294 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚499 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28249784948736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28249784948736 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28249766945792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31338648333312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚8448d<À¾&Ò¸M HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4097868 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚38 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30907650940928 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 _runtime‚ 359.795699911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 408917740294 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚499 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28249784948736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28249784948736 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28249766945792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31338648333312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410426945024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2307525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28660193890816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614&pZË0Ó+ _wandbruntime‚359  _runtime‚359Ð|éPÔÊ /loss/FRt2/assignment_loss‚7.387173488491872 /loss/FRt3/assignment_loss‚7.238859789712088 .loss/FRt1/detection_loss‚2.841053411907827 _step‚39 /loss/FRt1/assignment_loss‚7.228491530698888 0loss/FRt4/assignment_loss‚7.4543084325017155 1loss/SRqqt4/assignment_loss‚2.833045255054127 .loss/FRt2/detection_loss‚2.583095277674863 .loss/FBt1/detection_loss‚2.692418112023853 .loss/FBt3/detection_loss‚4.738319652922013 trainer/global_step‚499 2loss/SRqqt2/assignment_loss‚2.8153451054524155 2loss/SRqqt3/assignment_loss‚2.5690513864159583 0loss/FBt3/assignment_loss‚0.1922863873084733 &loss/total_loss‚101.86427058678845 epoch‚0 0loss/FBt1/assignment_loss‚0.5894241330182843 1loss/FBt4/assignment_loss‚0.40822785309781984 .loss/FRt4/detection_loss‚4.273888920610015 1loss/SRqqt4/detection_loss‚1.7744683666098375 2loss/SRqqt1/assignment_loss‚2.8727105422453447 /loss/FBt2/detection_loss‚21.881982939583914 % _timestamp‚1.7884482682426288e+09 0loss/SRqqt3/detection_loss‚4.114626301825046 _runtime‚ 359.797312008 1loss/SRqqt1/detection_loss‚1.8024640454695062 0loss/FBt2/assignment_loss‚0.4608458669889452 .loss/FRt3/detection_loss‚7.133690359336989 0loss/SRqqt2/detection_loss‚1.659083330239623 -loss/FBt4/detection_loss‚4.31941009759903Ðê7ÍTÕÎ 0loss/SRqqt3/detection_loss‚4.114626301825046 _runtime‚ 359.797312008 1loss/SRqqt1/detection_loss‚1.8024640454695062 0loss/FBt2/assignment_loss‚0.4608458669889452 .loss/FRt3/detection_loss‚7.133690359336989 0loss/SRqqt2/detection_loss‚1.659083330239623 -loss/FBt4/detection_loss‚4.31941009759903 /loss/FRt2/assignment_loss‚7.387173488491872 /loss/FRt3/assignment_loss‚7.238859789712088 .loss/FRt1/detection_loss‚2.841053411907827 _step‚39 /loss/FRt1/assignment_loss‚7.228491530698888 0loss/FRt4/assignment_loss‚7.4543084325017155 1loss/SRqqt4/assignment_loss‚2.833045255054127 .loss/FRt2/detection_loss‚2.583095277674863 .loss/FBt1/detection_loss‚2.692418112023853 .loss/FBt3/detection_loss‚4.738319652922013 trainer/global_step‚499 2loss/SRqqt2/assignment_loss‚2.8153451054524155 2loss/SRqqt3/assignment_loss‚2.5690513864159583 0loss/FBt3/assignment_loss‚0.1922863873084733 &loss/total_loss‚101.86427058678845 epoch‚0 0loss/FBt1/assignment_loss‚0.5894241330182843 1loss/FBt4/assignment_loss‚0.40822785309781984 .loss/FRt4/detection_loss‚4.273888920610015 1loss/SRqqt4/detection_loss‚1.7744683666098375 2loss/SRqqt1/assignment_loss‚2.8727105422453447 /loss/FBt2/detection_loss‚21.881982939583914 % _timestamp‚1.7884482682426288e+09'M·6 “Ö‚(:ˆ ŒœæÔÀæÕ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.024' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ôµ×‚(:Œ ŒœæÔùÔ°Ø disk.sda3.in‚0 proc.memory.rssMB‚ 2781.0625 cpu‚25.088572188243937 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.9765625% disk./.usageGB‚1108.3456268310547 disk.rbd2.out‚ 7.95166015625' disk.nvme0c0n1.out‚1178.65771484375$ memory_percent‚90.86220264434814 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 12.75390625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 12.75390625! disk.nvme0c0n1.in‚ 153.9765625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚1178.65771484375 network.recv‚216275 disk.nvme0n1.in‚ 153.9765625 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.0859375 disk.sda.in‚0) proc.memory.availableMB‚ 1497.13671875* proc.memory.percent‚16.974258422851562( disk./.usagePercent‚74.3820049805549% disk.nvme0n1.out‚1178.65771484375 disk.sda.out‚ 12.75390625 network.sent‚603174 disk.drbd1777.out‚09×0ƒ“Ø‚(:ˆ ›œæÔÑ£ÆÖ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.498' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±´‰qÙ‚(:… ›œæÔ°Á²Ø disk.dm-1.out‚13.09375 disk.sda3.in‚0 disk.sda3.out‚13.09375 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 153.9765625$ disk./.usageGB‚1108.345687866211 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.1484375 disk.nvme0n1.in‚ 153.9765625 network.sent‚652392% disk.nvme0n1.out‚1276.13525390625( proc.memory.availableMB‚ 1497.3984375" proc.memory.rssMB‚ 2781.0703125* proc.memory.percent‚16.974306106567383 disk.dm-0.in‚ 153.9765625" disk.dm-0.out‚1276.13525390625 disk.rbd3.out‚0 disk.sda.out‚13.09375 disk.rbd2.out‚ 9.58837890625 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1276.13525390625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚221898 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚90.86060523986816) disk./.usagePercent‚74.38200907667544 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.087750125860065 disk.sda2.in‚0ì´HížÚ‚(:“ ªœæÔ‚ôéÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.569' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.313799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÑGÛ‚(:‰ ªœæÔ°êÏØ disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.out‚1277.052734375$ memory_percent‚90.86062908172607* proc.memory.percent‚16.974306106567383 disk.dm-1.out‚ 14.00390625 disk.rbd1.out‚ 0.1484375 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 9.60400390625 network.sent‚658916) disk./.usagePercent‚74.38200907667544 disk.rbd3.in‚0 disk.drbd1777.out‚0$ disk./.usageGB‚1108.345687866211 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 14.00390625% disk.nvme0c0n1.out‚1277.052734375 cpu‚25.086359549089835 network.recv‚226565! disk.nvme0c0n1.in‚ 153.9765625 disk.nvme0n1.in‚ 153.9765625 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚1277.052734375 disk.dm-0.in‚ 153.9765625) proc.memory.availableMB‚ 1497.39453125" proc.memory.rssMB‚ 2781.0703125 disk.sda.out‚ 14.00390625 proc.cpu.threads‚17«wÅ0Ü+ _wandbruntime‚394  _runtime‚394TD£Ãݽ _runtime‚ 394.471609817 _step‚40 & lr-AdamW/pg1‚8.368248162895699e-05 & lr-AdamW/pg2‚8.368248162895699e-05 trainer/global_step‚549 $ _timestamp‚1.788448302917138e+09ÛkŸÇÞÁ & lr-AdamW/pg2‚8.368248162895699e-05 trainer/global_step‚549 $ _timestamp‚1.788448302917138e+09 _runtime‚ 394.471609817 _step‚40 & lr-AdamW/pg1‚8.368248162895699e-05(’ü~0ß+ _wandbruntime‚394  _runtime‚394‘å„´'à®O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 394.472613047 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 450593323008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30661891841154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚41 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10181718 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884483029179919e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2534824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 448936644738 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚549 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31011065849344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7716396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚4309©bÇ0 á²O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 450593323008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30661891841154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚41 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10181718 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884483029179919e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2534824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 448936644738 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚549 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31011065849344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7716396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚ 394.472613047)ï®±W0â+ _wandbruntime‚395  _runtime‚395Ì.²û¾&ã¸M O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 % _timestamp‚1.7884483035809405e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚42 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31067436891136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 474350849536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30718283876454 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152¡N*Â&ä¼M JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 450407510118 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31067418888192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 % _timestamp‚1.7884483035809405e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 474196957696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚42 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31067436891136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 474350849536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10200172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30718283876454 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 516815502336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32958178763776 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 490426414354 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33448605178130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34320970519040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2766960 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 516961018880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430×^YÓ¸'ñ²O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33448605178130 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34320970519040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2766960 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 516961018880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884483386849942e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7673462 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 516815502336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11107770 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32958178763776 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 490426414354 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0-¦¦&ù0ò+ _wandbruntime‚430  _runtime‚430ùp­¾&ó¸M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 492398556672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33885070830592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 _step‚46 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34378142999040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884483393483193e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33885088833536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37072748647424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 517853663744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚599 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 491242849526 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 517701869056 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33885070830592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33013065854976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 430.903334649 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33885088833536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F®e¼ô¼M _step‚46 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34378142999040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884483393483193e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33885088833536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37072748647424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 517853663744 REDeviceStaæ3g3 tsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚599 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 491242849526 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 517701869056 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33885070830592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33013065854976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 430.903334649 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33885088833536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 492398556672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33885070830592 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004.Ü8<0õ+ _wandbruntime‚430  _runtime‚430û\9ÄOöÉ /loss/FBt3/detection_loss‚3.8156576922961642 epoch‚0 _runtime‚ 430.904533357 /loss/FRt3/assignment_loss‚7.240461139678955 0loss/SRqqt2/detection_loss‚2.059934709018663 _step‚47 /loss/FRt1/detection_loss‚3.1225228256412914 0loss/SRqqt1/detection_loss‚1.396848381916564 2loss/SRqqt2/assignment_loss‚2.5297857376112454 /loss/FRt2/detection_loss‚2.0447574251809635 $loss/total_loss‚97.5674917333574 0loss/FRt2/assignment_loss‚6.7589700737515015 /loss/FRt4/assignment_loss‚7.352170679304335 2loss/SRqqt1/assignment_loss‚2.6354197789640987 0loss/FBt2/assignment_loss‚0.3858746395599155 -loss/FRt4/detection_loss‚4.25658303913143 0loss/SRqqt3/detection_loss‚4.749132056127895 .loss/FBt1/detection_loss‚3.248407521833494 /loss/FRt1/assignment_loss‚7.184726218787992 1loss/SRqqt3/assignment_loss‚2.863706617644339 .loss/FBt4/detection_loss‚7.287195571740582 % _timestamp‚1.7884483393502102e+09 .loss/FRt3/detection_loss‚2.851285009384155 1loss/SRqqt4/detection_loss‚3.6244138768354848 1loss/SRqqt4/assignment_loss‚3.188693296164274 1loss/FBt4/assignment_loss‚0.22014299445700916 /loss/FBt2/detection_loss‚17.917580991983414 trainer/global_step‚599 0loss/FBt1/assignment_loss‚0.5472062919411416 0loss/FBt3/assignment_loss‚0.2860151644024968ÿ ñS÷Í 1loss/SRqqt3/assignment_loss‚2.863706617644339 .loss/FBt4/detection_loss‚7.287195571740582 % _timestamp‚1.7884483393502102e+09 .loss/FRt3/detection_loss‚2.851285009384155 1loss/SRqqt4/detection_loss‚3.6244138768354848 1loss/SRqqt4/assignment_loss‚3.188693296164274 1loss/FBt4/assignment_loss‚0.22014299445700916 /loss/FBt2/detection_loss‚17.917580991983414 trainer/global_step‚599 0loss/FBt1/assignment_loss‚0.5472062919411416 0loss/FBt3/assignment_loss‚0.2860151644024968 /loss/FBt3/detection_loss‚3.8156576922961642 epoch‚0 _runtime‚ 430.904533357 /loss/FRt3/assignment_loss‚7.240461139678955 0loss/SRqqt2/detection_loss‚2.059934709018663 _step‚47 /loss/FRt1/detection_loss‚3.1225228256412914 0loss/SRqqt1/detection_loss‚1.396848381916564 2loss/SRqqt2/assignment_loss‚2.5297857376112454 /loss/FRt2/detection_loss‚2.0447574251809635 $loss/total_loss‚97.5674917333574 0loss/FRt2/assignment_loss‚6.7589700737515015 /loss/FRt4/assignment_loss‚7.352170679304335 2loss/SRqqt1/assignment_loss‚2.6354197789640987 0loss/FBt2/assignment_loss‚0.3858746395599155 -loss/FRt4/detection_loss‚4.25658303913143 0loss/SRqqt3/detection_loss‚4.749132056127895 .loss/FBt1/detection_loss‚3.248407521833494 /loss/FRt1/assignment_loss‚7.184726218787992/’_ hžø‚(:“ לæÔÔêÓÕ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.622999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìm²ù‚(:’ לæÔ˜‚ˆØ% disk.nvme0n1.out‚1472.34521484375% disk./.usageGB‚1108.3456916809082 disk.rbd3.in‚0 disk.sda3.out‚ 15.38671875 disk.rbd2.out‚ 9.7685546875 disk.dm-1.out‚ 15.38671875 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 153.9765625' disk.nvme0c0n1.out‚1472.34521484375) proc.memory.availableMB‚ 1497.22265625) disk./.usagePercent‚74.38200933268297 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.24365234375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.9765625 network.sent‚720524 disk.rbd1.in‚0 disk.sda.out‚ 15.38671875 cpu‚25.081785657077273 proc.cpu.threads‚17 disk.drbd1777.in‚0* proc.memory.percent‚16.974353790283203! proc.memory.rssMB‚ 2781.078125 disk.dm-0.in‚ 153.9765625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0$ memory_percent‚90.86167812347412 network.recv‚240932" disk.dm-0.out‚1472.34521484375 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0ðï\â’ú‚(:‡ æœæÔ—ýçÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁHY û‚(:‚ æœæÔÄÝ‹Ø) disk./.usagePercent‚74.38198731603504 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 9.7841796875! disk.nvme0c0n1.in‚ 153.9765625% disk.nvme0n1.out‚1472.96435546875 disk.sda2.out‚0 network.recv‚246732 disk.drbd1777.out‚0 disk.sda2.in‚0 network.sent‚770527 disk.rbd1.in‚0 disk.sda.out‚15.9375" disk.dm-0.out‚1472.96435546875# memory_percent‚90.8625602722168 disk.rbd2.in‚0 disk.dm-1.in‚0 cpu‚25.08078028436313 proc.cpu.threads‚17% disk./.usageGB‚1108.3453636169434 disk.rbd1.out‚ 0.24365234375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚15.9375 disk.rbd3.in‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚1472.96435546875 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.9765625' proc.memory.availableMB‚ 1497.078125* proc.memory.percent‚16.974353790283203 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 153.9765625 disk.dm-1.out‚15.9375! proc.memory.rssMB‚ 2781.078125¼ßn“ü‚(:ˆ õœæÔã“ÌÕ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]3 ý‚(:û õœæÔŸÇØ network.recv‚251211 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 network.sent‚776526 disk.rbd3.in‚0 disk.sda3.out‚ 16.49609375 disk.rbd0.out‚0 disk.dm-0.out‚ 1570.4375! proc.memory.rssMB‚ 2781.078125* proc.memory.percent‚16.974353790283203 cpu‚25.079834890241592% disk./.usageGB‚1108.3454856872559 disk.sda.out‚ 16.49609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 11.4208984375 disk.nvme0n1.out‚ 1570.4375 disk.drbd1777.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 153.9765625 disk.nvme0n1.in‚ 153.9765625 disk.dm-0.in‚ 153.9765625 disk.dm-1.in‚0) proc.memory.availableMB‚ 1497.07421875 disk.sda3.in‚0) disk./.usagePercent‚74.38199550827613 disk.nvme0c0n1.out‚ 1570.4375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 16.49609375 disk.rbd3.out‚0 disk.rbd2.in‚0# memory_percent‚90.8625841140747 proc.cpu.threads‚17¶‡„b0þ+ _wandbruntime‚465  _runtime‚465ºg#Äÿ¾ & lr-AdamW/pg1‚9.892519230818411e-05 & lr-AdamW/pg2‚9.892519230818411e-05 trainer/global_step‚649 % _timestamp‚1.7884483739489014e+09 _runtime‚ 465.503364667 _step‚48Ir¨qȀ & lr-AdamW/pg2‚9.892519230818411e-05 trainer/global_step‚649 % _timestamp‚1.7884483739489014e+09 _runtime‚ 465.503364667 _step‚48 & lr-AdamW/pg1‚9.892519230818411e-050µ}E0+ _wandbruntime‚465  _runtime‚465!Â(ë´'‚®O % _timestamp‚1.7884483739496264e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21151818 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 _step‚49 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36646369734144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8312520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12033822 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35703368252416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35703385291776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 465.504251418 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9117996 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37178934668800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12052362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40093610852864 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2997618 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5314902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 530607323554 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 trainer/global_step‚649d*I%ƒ²O E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 trainer/global_step‚649 % _timestamp‚1.7884483739496264e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21151818 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 _step‚49 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36646369734144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8312520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12033822 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35703368252416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35703385291776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 465.504251418 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9117996 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37178934668800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12052362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40093610852864 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2997618 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5314902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 530607323554 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚01Šö40„+ _wandbruntime‚466  _runtime‚466Œ|ë¾&…¸M NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9132014 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36289696141494 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12052276 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40155285530624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37236780747264 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8325307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36702740775936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 _step‚50 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36702740775936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37236107135488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚649 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40155261590528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884483746116054e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21202856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5323068 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0<ÔÅÂ&†¼M TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21202856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5323068 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9132014 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36289696141494 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12052276 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40155285530624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37236780747264 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8325307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36702740775936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 _step‚50 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36702740775936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37236107135488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚649 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40155261590528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884483746116054e+092¹—}0‡+ _wandbruntime‚466  _runtime‚466ì_éMˆÇ /loss/FRt3/assignment_loss‚6.599203912835372 1loss/SRqqt1/detection_loss‚1.8855020199501316 .loss/FBt4/detection_loss‚4.278262530054365 % _timestamp‚1.7884483746140363e+09 /loss/FRt2/assignment_loss‚6.671226978302002 1loss/FBt3/assignment_loss‚0.31769129529090734 .loss/FRt3/detection_loss‚4.236459248944333 /loss/FBt2/detection_loss‚22.476492231090862 .loss/FRt1/detection_loss‚3.076880510554326 0loss/FBt2/assignment_loss‚0.5728296366360155 .loss/FRt2/detection_loss‚2.641104915046266 -loss/FBt3/detection_loss‚5.64083456248045 trainer/global_step‚649 2loss/SRqqt2/assignment_loss‚2.5876028465502188 /loss/FRt4/detection_loss‚3.6006431601396423 0loss/SRqqt4/detection_loss‚2.427099869996382 .loss/FBt1/detection_loss‚1.162605787763649 /loss/FRt1/assignment_loss‚7.085851075697918 /loss/FRt4/assignment_loss‚7.039360604635099 0loss/SRqqt3/detection_loss‚5.530664448227201 _runtime‚ 466.168414059 _step‚51 0loss/SRqqt3/assignment_loss‚2.78176606127194 1loss/SRqqt4/assignment_loss‚2.970140478190254 0loss/FBt1/assignment_loss‚0.6588793698705026 0loss/FBt4/assignment_loss‚0.2079607153427787 epoch‚0 2loss/SRqqt1/assignment_loss‚2.5242215799073042 1loss/SRqqt2/detection_loss‚2.1947757055926504 %loss/total_loss‚99.16805954437058žR%;Q‰Ë 0loss/SRqqt3/assignment_loss‚2.78176606127194 1loss/SRqqt4/assignment_loss‚2.970140478190254 0loss/FBt1/assignment_loss‚0.6588793698705026 0loss/FBt4/assignment_loss‚0.2079607153427787 epoch‚0 2loss/SRqqt1/assignment_loss‚2.5242215799073042 1loss/SRqqt2/detection_loss‚2.1947757055926504 %loss/total_loss‚99.16805954437058 /loss/FRt3/assignment_loss‚6.599203912835372 1loss/SRqqt1/detection_loss‚1.8855020199501316 .loss/FBt4/detection_loss‚4.278262530054365 % _timestamp‚1.7884483746140363e+09 /loss/FRt2/assignment_loss‚6.671226978302002 1loss/FBt3/assignment_loss‚0.31769129529090734 .loss/FRt3/detection_loss‚4.236459248944333 /loss/FBt2/detection_loss‚22.476492231090862 .loss/FRt1/detection_loss‚3.076880510554326 0loss/FBt2/assignment_loss‚0.5728296366360155 .loss/FRt2/detection_loss‚2.641104915046266 -loss/FBt3/detection_loss‚5.64083456248045 trainer/global_step‚649 2loss/SRqqt2/assignment_loss‚2.5876028465502188 /loss/FRt4/detection_loss‚3.6006431601396423 0loss/SRqqt4/detection_loss‚2.427099869996382 .loss/FBt1/detection_loss‚1.162605787763649 /loss/FRt1/assignment_loss‚7.085851075697918 /loss/FRt4/assignment_loss‚7.039360604635099 0loss/SRqqt3/detection_loss‚5.530664448227201 _runtime‚ 466.168414059 _step‚513rv8“Š‚(:ˆ „æÔßÞêÕ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿr†6‹‚(:‰ „æÔÚÖ¿Ø# disk.nvme0n1.out‚1571.294921875 disk.drbd1777.out‚0 disk.rbd2.out‚ 11.4326171875 disk.sda1.out‚0% disk.nvme0c0n1.out‚1571.294921875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 16.93359375 cpu‚25.07940623347692 disk.sda.in‚0% disk./.usageGB‚1108.3454856872559 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚256390( proc.memory.availableMB‚ 1497.1015625 disk.sda3.in‚0 disk.sda3.out‚ 16.93359375 disk.nvme0n1.in‚ 153.9765625 disk.dm-0.in‚ 153.9765625 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625$ memory_percent‚90.86241722106934 proc.cpu.threads‚17 disk.rbd0.out‚0) disk./.usagePercent‚74.38199550827613! disk.nvme0c0n1.in‚ 153.9765625 disk.dm-0.out‚1571.294921875 disk.sda2.in‚0 disk.dm-1.out‚ 16.93359375 disk.rbd3.out‚0! proc.memory.rssMB‚ 2781.078125 network.sent‚783299 disk.rbd0.in‚0* proc.memory.percent‚16.974353790283203Z¥åPžŒ‚(:“ “æÔ’Ø°Õ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.337' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.267399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ-Q˜ ‚(:€ “æÔºóùØ% disk./.usageGB‚1108.3454856872559 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚17.3125 network.recv‚261937 disk.rbd2.in‚0% disk.nvme0c0n1.out‚1668.677734375 disk.drbd1777.in‚0) proc.memory.availableMB‚ 1497.13671875* proc.memory.percent‚16.974353790283203 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 153.9765625 disk.dm-0.out‚1668.677734375$ memory_percent‚90.86220264434814 disk.sda.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 cpu‚25.078034371932247 network.sent‚832541) disk./.usagePercent‚74.38199550827613 disk.rbd3.out‚0 disk.sda.out‚17.3125 disk.dm-0.in‚ 153.9765625! proc.memory.rssMB‚ 2781.078125 proc.cpu.threads‚17 disk.rbd2.out‚11.51904296875 disk.nvme0n1.in‚ 153.9765625# disk.nvme0n1.out‚1668.677734375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚17.3125dƒ0Ž+ _wandbruntime‚501  _runtime‚5014{^ðľ _step‚52 & lr-AdamW/pg1‚0.0001065465476477977 & lr-AdamW/pg2‚0.0001065465476477977 trainer/global_step‚699 % _timestamp‚1.7884484097348928e+09 _runtime‚ 501.289406405Ác5ûÈ & lr-AdamW/pg1‚0.0001065465476477977 & lr-AdamW/pg2‚0.0001065465476477977 trainer/global_step‚699 % _timestamp‚1.7884484097348928e+09 _runtime‚ 501.289406405 _step‚524¡Îš0‘+ _wandbruntime‚501  _runtime‚501î;aeC ’®O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9818796 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9818798 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 571442662962 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 573550740480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43779620863488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43176147736064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚53'ü¸'“²O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9818798 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38448591819776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12978414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39464021676544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5723302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39464021676544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9818796 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603449187328 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 571442662962 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 573550740480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43779620863488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43176147736064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚53 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98187965îè0”+ _wandbruntime‚501  _runtime‚501¶‚¼&•¶M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12996894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 _step‚54 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3235357 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 575025777152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43237798473728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350›^f‘È–ºM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39520392718336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 574370168320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43237798473728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5731470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12996894 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40094744883712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_enû’µûød/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38503478910976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12978328 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 _step‚54 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3235357 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39520374715392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 575025777152 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚749 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10519596 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 646587082240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 $ _timestamp‚1.788448445415835e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46905115699200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 614536546304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3462812 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46905295641600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _runtime‚ 536.970498756 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10519598 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10519598 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13885926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚57 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24405522 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10519598 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13885926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚57 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41806059310786 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24405522 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚749 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 612278002370 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10519596 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6465870822409¥½0¤+ _wandbruntime‚537  _runtime‚537ª±È:×¥¼M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41861779876310 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884484461073608e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13904380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.lar¶ìÝ{ëge_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46967630907392 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46967815030784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚749 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚58 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13922946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42954056243712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46320335356928è'#Æ&¦ÀM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13904380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 613748867750 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 613094437334 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42954056243712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 537.662148174 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6139868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42338044660736 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 647295550464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41248702478336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10533612 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9607350 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13922946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42954056243712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46320335356928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42338026657792 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41861779876310 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884484461073608e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13904380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46967630907392 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46967815030784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚749 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚58 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F¦–‚“®‚(:ˆ ÞæÔ¬ë±Ö gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾]„¯‚(:Ž ÞæÔ¶¯Ù disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 21.18359375* proc.memory.percent‚16.974425315856934 network.sent‚951271# disk.nvme0n1.out‚1865.720703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2781.08984375 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚1865.720703125 disk.dm-0.out‚1865.720703125 disk.sda3.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.9765625) proc.memory.availableMB‚ 1496.46484375 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 disk.rbd1.in‚0 disk.sda3.out‚ 21.18359375$ memory_percent‚90.86630344390869 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.out‚0 network.recv‚287410) disk./.usagePercent‚74.38198347592203 disk.dm-1.in‚0 disk.dm-1.out‚ 21.18359375 cpu‚25.074467665626184 proc.cpu.threads‚17% disk./.usageGB‚1108.3453063964844 disk.rbd2.out‚13.32763671875! disk.nvme0c0n1.in‚ 153.9765625 disk.rbd3.in‚0©pl%0°+ _wandbruntime‚572  _runtime‚572=ÓÊ2ƱÀ ' lr-AdamW/pg1‚0.00012178925832702483 ' lr-AdamW/pg2‚0.00012178925832702483 trainer/global_step‚799 % _timestamp‚1.7884484808956885e+09 _runtime‚ 572.450087742 _step‚60•ÝluÊ²Ä ' lr-AdamW/pg2‚0.00012178925832702483 trainer/global_step‚799 % _timestamp‚1.7884484808956885e+09 _runtime‚ 572.450087742 _step‚60 ' lr-AdamW/pg1‚0.00012178925832702483<mçž0³+ _wandbruntime‚572  _runtime‚572~)Ï´´O trainer/global_step‚799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 655522352128 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10234947 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 689571738624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45754847913472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44592772647762 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14830518 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50030793241088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.curW_ö)¡ rent‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11220396 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 _step‚61 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6540100 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 656177947648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 _runtime‚ 572.451120898 I£¾'µ¸O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14830518 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11220398 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44592101178194 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 158099456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44592772647762 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14830518 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10234479 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50030793241088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11220396 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 _step‚61 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6540100 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 656177947648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 _runtime‚ 572.451120898 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 655522352128 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10234947 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 689571738624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50030975280640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45754847913472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14811978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654584207158 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 573.117052906 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11234412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 656997388800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14830432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 656997388800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45812693991936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45812020380160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50093319687680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26064844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49402872240128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 _step‚62 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14830432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26083412 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 656341779968˜ôxŠ$¸ÂM % _timestamp‚1.7884484815619898e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14848998 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26064844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49402872240128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10247713 _step‚62 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14830432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26083412 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 656341779968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45155678600192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3698982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654584207158 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 573.117052906 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11234412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 656997388800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14830432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 656997388800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45812693991936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179929088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653929776742 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45812020380160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50093319687680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 690603436544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 Ot.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11234414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 690447447552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844>¨œÚ0¹+ _wandbruntime‚573  _runtime‚573ð–\!TºÎ 1loss/SRqqt3/assignment_loss‚2.114529819714583 0loss/FBt3/assignment_loss‚0.2627158132707336 /loss/FRt1/detection_loss‚3.5085854193393877 .loss/FBt4/detection_loss‚12.72651403186529 epoch‚0 0loss/FBt1/assignment_loss‚0.6250458719829718 _step‚63 /loss/FRt2/detection_loss‚1.9740897658970449 /loss/FRt3/detection_loss‚3.4590895113192106 1loss/SRqqt1/detection_loss‚3.5109649138017134 trainer/global_step‚799 /loss/FRt3/assignment_loss‚6.608457239050614 /loss/FRt4/assignment_loss‚6.242348364421299 2loss/SRqqt1/assignment_loss‚2.9951760281216013 1loss/FBt4/assignment_loss‚0.16711247770581394 .loss/FBt2/detection_loss‚7.103188740856507 /loss/FRt2/assignment_loss‚4.479309630084347 1loss/SRqqt2/assignment_loss‚2.471386507304028 1loss/SRqqt3/detection_loss‚3.2448465011336585 0loss/SRqqt4/detection_loss‚4.486401898662249 % _timestamp‚1.7884484815638459e+09 _runtime‚ 573.118172725 /loss/FBt3/detection_loss‚3.0361013013455604 /loss/FRt1/assignment_loss‚7.135632128328891 2loss/SRqqt4/assignment_loss‚2.9336368362108867 /loss/FBt1/detection_loss‚10.185098864138126 $loss/total_loss‚94.3843497332291 1loss/FBt2/assignment_loss‚0.45661104688812937 /loss/FRt4/detection_loss‚3.3877845881950286 1loss/SRqqt2/detection_loss‚1.2697224335914308)¢/X»Ò /loss/FRt3/assignment_loss‚6.608457239050614 /loss/FRt4/assignment_loss‚6.242348364421299 2loss/SRqqt1/assignment_loss‚2.9951760281216013 1loss/FBt4/assignment_loss‚0.16711247770581394 .loss/FBt2/detection_loss‚7.103188740856507 /loss/FRt2/assignment_loss‚4.479309630084347 1loss/SRqqt2/assignment_loss‚2.471386507304028 1loss/SRqqt3/detection_loss‚3.2448465011336585 0loss/SRqqt4/detection_loss‚4.486401898662249 % _timestamp‚1.7884484815638459e+09 _runtime‚ 573.118172725 /loss/FBt3/detection_loss‚3.0361013013455604 /loss/FRt1/assignment_loss‚7.135632128328891 2loss/SRqqt4/assignment_loss‚2.9336368362108867 /loss/FBt1/detection_loss‚10.185098864138126 $loss/total_loss‚94.3843497332291 1loss/FBt2/assignment_loss‚0.45661104688812937 /loss/FRt4/detection_loss‚3.3877845881950286 1loss/SRqqt2/detection_loss‚1.2697224335914308 1loss/SRqqt3/assignment_loss‚2.114529819714583 0loss/FBt3/assignment_loss‚0.2627158132707336 /loss/FRt1/detection_loss‚3.5085854193393877 .loss/FBt4/detection_loss‚12.72651403186529 epoch‚0 0loss/FBt1/assignment_loss‚0.6250458719829718 _step‚63 /loss/FRt2/detection_loss‚1.9740897658970449 /loss/FRt3/detection_loss‚3.4590895113192106 1loss/SRqqt1/detection_loss‚3.5109649138017134 trainer/global_step‚799?U]ÅR“¼‚(:ˆ íæÔÚÛãÕ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ ‚½‚(:‡ íæÔ…ÜžØ% disk./.usageGB‚1108.3454093933105 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 153.9765625 disk.nvme0n1.in‚ 153.9765625 disk.dm-1.in‚0 cpu‚25.073962527107778 disk.sda1.in‚0 network.sent‚957374# disk.nvme0n1.out‚1963.220703125 disk.sda2.out‚0 disk.rbd3.in‚0* proc.memory.percent‚16.974425315856934 proc.cpu.threads‚17) disk./.usagePercent‚74.38199038812544 disk.sda2.in‚0 disk.dm-1.out‚ 21.6015625 disk.rbd1.out‚ 0.306640625 disk.rbd2.out‚ 13.4140625 disk.sda.out‚ 21.6015625 disk.sda3.in‚0$ memory_percent‚90.86711406707764% disk.nvme0c0n1.out‚1963.220703125 disk.sda.in‚0 network.recv‚292228 disk.dm-0.in‚ 153.9765625 disk.rbd1.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1496.33203125# proc.memory.rssMB‚ 2781.08984375 disk.rbd0.out‚0 disk.dm-0.out‚1963.220703125 disk.sda3.out‚ 21.6015625Ý ÆÞž¾‚(:“ üæÔô÷¬Ö gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.760199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mòa ¿‚(:” üæÔ½æÔØ disk.drbd1777.out‚0 disk.dm-1.out‚ 22.44140625 disk.sda3.out‚ 22.44140625) disk./.usagePercent‚74.38199192417065 disk.rbd1.out‚ 0.306640625( proc.memory.availableMB‚ 1496.3046875 disk.sda1.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚1963.74658203125 disk.rbd3.in‚0* proc.memory.percent‚16.974425315856934 disk.sda2.out‚0 network.recv‚298582 disk.sda.in‚0 disk.dm-0.in‚ 153.9765625$ memory_percent‚90.86728096008301 cpu‚25.07335259469862 disk.sda2.in‚0% disk./.usageGB‚1108.3454322814941 disk.rbd1.in‚0 disk.sda.out‚ 22.44140625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.9765625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2781.08984375 disk.sda3.in‚0 network.sent‚ 1.007878e+06 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 13.4296875% disk.nvme0n1.out‚1963.74658203125 disk.drbd1777.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 153.9765625' disk.nvme0c0n1.out‚1963.74658203125FÑ ˆ0À+ _wandbruntime‚608  _runtime‚608¥ 6ÄÁ¾ _runtime‚ 608.055970097 _step‚64 & lr-AdamW/pg1‚0.0001294106136666384 & lr-AdamW/pg2‚0.0001294106136666384 trainer/global_step‚849 % _timestamp‚1.7884485165015764e+09vZÈ & lr-AdamW/pg1‚0.0001294106136666384 & lr-AdamW/pg2‚0.0001294106136666384 trainer/global_step‚849 % _timestamp‚1.7884485165015764e+09 _runtime‚ 608.055970097 _step‚64@éÌ‚30Ã+ _wandbruntime‚608  _runtime‚608ÓxOŸ'IJO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47916977503744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6948502 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 # _timestamp‚1.78844851650234e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 _step‚65 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3927876 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46684194364416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47916977503744 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 608.057061514 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11921196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6948500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 696508157952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 _step‚65 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3927876 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46684194364416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47916995506688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47916977503744 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53156795469312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 608.057061514 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11921196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6948500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 696508157952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15775050 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15775050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6956668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚849 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 608.715009934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 UGDeviceStatsMonitor.on_train_batch_end/reserved_†·r‡@bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚66 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697327585792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3932067 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697327585792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3932067 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15775050 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47973330542592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52485409123328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15775050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6956668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚849 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 608.715009934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚66 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566B‰Z0É+ _wandbruntime‚608  _runtime‚608àÁÍsNÊÈ /loss/FRt4/assignment_loss‚6.404727112163197 1loss/SRqqt4/assignment_loss‚3.069208989972654 epoch‚0 % _timestamp‚1.7884485171620464e+09 2loss/SRqqt1/assignment_loss‚3.1280996568741335 1loss/SRqqt3/assignment_loss‚1.994082476331019 1loss/FBt2/assignment_loss‚0.31962523838172696 /loss/FRt1/detection_loss‚3.0773054117506202 /loss/FRt2/assignment_loss‚4.059710383415222 /loss/FRt3/assignment_loss‚6.235898863185536 /loss/FRt3/detection_loss‚2.5920780897140503 1loss/SRqqt3/detection_loss‚3.6893799963750338 /loss/FRt1/assignment_loss‚6.658848957581953 /loss/FRt2/detection_loss‚2.0305203565529415 0loss/SRqqt4/detection_loss‚5.227256761944813 trainer/global_step‚849 _step‚67 0loss/FBt4/assignment_loss‚0.7338084337821379 /loss/SRqqt1/detection_loss‚4.57671164985626 1loss/SRqqt2/detection_loss‚1.3187107629459964 .loss/FBt2/detection_loss‚4.579189678797355 /loss/FBt3/detection_loss‚2.6627195527156196 0loss/FBt1/assignment_loss‚0.6962951120572792 .loss/FRt4/detection_loss‚3.221549378877336 .loss/FBt4/detection_loss‚1.614602429131035 _runtime‚ 608.716464174 1loss/SRqqt2/assignment_loss‚2.496725313611083 0loss/FBt3/assignment_loss‚0.3932161546412563 -loss/FBt1/detection_loss‚6.98891661635467 %loss/total_loss‚77.76918737701294Ò_³DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 738149559296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16682622 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50164184913010 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51472797008384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚69 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55506319208960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50734629446144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51472797008384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55506295268864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0é ô¸Õ¸O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50164856382578 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49429400892416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 643.498532521 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 776090788864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4159801 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 MFDeviceStatsMonitor.on_t?u‚Ù rain_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 738149559296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16682622 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50164184913010 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51472797008384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚69 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55506319208960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50734629446144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11517179 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51472797008384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55506295268864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11516701 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360E]7µÆ0Ö+ _wandbruntime‚644  _runtime‚644½ÊÒÄ&×¾M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49484322062336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16682536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50790982484992 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 738313391616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 738969000448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55567946006528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49484305022976 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 735600455558 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 644.164065373 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884485526089082e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚70 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51529969488384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4164924 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50791000487936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 738969000448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12636014 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50791000487936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16682536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16701102 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 736254885974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51529295876608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 644.164065373 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884485526089082e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚70 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51529969488384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4164924 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50791000487936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 738969000448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12636014 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50791000487936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16682536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16701102 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 736254885974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51529295876608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53552281388544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779135365120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4391185 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53552281388544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775619360002 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17608674 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12156984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30912930 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53552299391488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 819221993984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17590134 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54331434756608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 679.653335129 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58588856092160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884485880984743e+09 _step‚73 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778479769600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13322798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844Ì–Ùq¾'ç¸O M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 819221993984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17590134 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54331434756608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 679.653335129 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58588856092160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884485880984743e+09 _step‚73 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778479769600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13322798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58588832152064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53552281388544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779135365120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4391185 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53552281388544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775619360002 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17608674 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12156984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30912930 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53552299391488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096IËe0è+ _wandbruntime‚680  _runtime‚680wøã$é½M K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 trainer/global_step‚949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 _runtime‚ 680.323798695 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 $ _timestamp‚1.788448588768739e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚74 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMoniå„з¼tor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 $ _timestamp‚1.788448588768739e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53608652430336 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚74 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53608634427392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 trainer/global_step‚949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 _runtime‚ 680.323798695 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779299197440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 _runtime‚ 715.369197534 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0¿´‹,6÷¸O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57189398906368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55736268647826 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 820121170944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 _runtime‚ 715.369197534 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18534726 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 % _timestamp‚1.7884486238145888e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32539782 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18516186 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14023596 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57190072504832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56369951333888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56369933330944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14023598 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12798119 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464Mì¥ì‡0ø+ _wandbruntime‚716  _runtime‚716çì Â&ù¼M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8181870 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174863360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54974718078976 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 863240859136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18553206 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57247244984832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56426286369792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62596260632064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61733019772928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14037614 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57247244984832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56426286369792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18534640 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817271134374 $ _timestamp‚1.788448624498205e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14037614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 820285003264 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 _runtime‚ 716.05307158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57247244984832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198803456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56426286369792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18534640 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 820940612096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817271134374 $ _timestamp‚1.788448624498205e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14037614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56426304372736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 820285003264 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 _runtime‚ 716.05307158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56426304372736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12810892 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57247244984832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56426286369792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32572252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62596260632064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61733019772928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14037614 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549NC¸g<0û+ _wandbruntime‚716  _runtime‚716ÕÔNÁOüÉ % _timestamp‚1.7884486244999127e+09 1loss/SRqqt2/assignment_loss‚2.714604757048867 1loss/FBt1/assignment_loss‚0.36143197181324166 1loss/FBt2/assignment_loss‚0.15245896492588243 .loss/FRt4/detection_loss‚2.442805945735287 0loss/SRqqt3/detection_loss‚3.326366150849744 _runtime‚ 716.054219283 0loss/FRt2/assignment_loss‚3.5012375861406326 .loss/FRt3/assignment_loss‚6.06973032951355 1loss/SRqqt2/detection_loss‚1.6572190393397415 %loss/total_loss‚75.73985340067799 1loss/SRqqt1/assignment_loss‚2.427044055219424 /loss/FRt1/detection_loss‚3.0803977211839273 .loss/FBt1/detection_loss‚8.441894501447678 .loss/FBt4/detection_loss‚8.382592919591163 trainer/global_step‚999 /loss/FRt4/assignment_loss‚5.945008666427047 2loss/SRqqt4/assignment_loss‚2.9613326125674777 /loss/FBt2/detection_loss‚1.8311707815307232 epoch‚0 _step‚79 /loss/FRt1/assignment_loss‚4.797991338529084 .loss/FRt2/detection_loss‚2.330358790466562 .loss/FBt3/detection_loss‚2.751248885833082 1loss/SRqqt3/assignment_loss‚1.992231816984713 0loss/FBt3/assignment_loss‚0.4337804209781206 1loss/SRqqt1/detection_loss‚1.9351760820817139 0loss/SRqqt4/detection_loss‚4.877130318019125 0loss/FBt4/assignment_loss‚0.6965491382385759 .loss/FRt3/detection_loss‚2.630090606212616ÓÙ0 SýÍ epoch‚0 _step‚79 /loss/FRt1/assignment_loss‚4.797991338529084 .loss/FRt2/detection_loss‚2.330358790466562 .loss/FBt3/detection_loss‚2.751248885833082 1loss/SRqqt3/assignment_loss‚1.992231816984713 0loss/FBt3/assignment_loss‚0.4337804209781206 1loss/SRqqt1/detection_loss‚1.9351760820817139 0loss/SRqqt4/detection_loss‚4.877130318019125 0loss/FBt4/assignment_loss‚0.6965491382385759 .loss/FRt3/detection_loss‚2.630090606212616 % _timestamp‚1.7884486244999127e+09 1loss/SRqqt2/assignment_loss‚2.714604757048867 1loss/FBt1/assignment_loss‚0.36143197181324166 1loss/FBt2/assignment_loss‚0.15245896492588243 .loss/FRt4/detection_loss‚2.442805945735287 0loss/SRqqt3/detection_loss‚3.326366150849744 _runtime‚ 716.054219283 0loss/FRt2/assignment_loss‚3.5012375861406326 .loss/FRt3/assignment_loss‚6.06973032951355 1loss/SRqqt2/detection_loss‚1.6572190393397415 %loss/total_loss‚75.73985340067799 1loss/SRqqt1/assignment_loss‚2.427044055219424 /loss/FRt1/detection_loss‚3.0803977211839273 .loss/FBt1/detection_loss‚8.441894501447678 .loss/FBt4/detection_loss‚8.382592919591163 trainer/global_step‚999 /loss/FRt4/assignment_loss‚5.945008666427047 2loss/SRqqt4/assignment_loss‚2.9613326125674777 /loss/FBt2/detection_loss‚1.8311707815307232O…^ žþ‚(:“ ôžæÔ†»ÉÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.374799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¼+q ÿ‚(:• ôžæÔ•…­Ø disk.sda3.out‚ 26.9140625 disk.rbd2.in‚0' disk.nvme0c0n1.out‚2361.05419921875 disk.sda.out‚ 26.9140625 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.974472999572754 network.sent‚ 1.185222e+06" disk.dm-0.out‚2361.05419921875 disk.rbd3.in‚0% disk.nvme0n1.out‚2361.05419921875 disk.dm-1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2781.09765625 network.recv‚336063 disk.rbd1.out‚ 0.306640625 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.06894564409184 proc.cpu.threads‚17) disk./.usagePercent‚74.38234009441722% disk./.usageGB‚1108.3506202697754 disk.rbd2.out‚17.02392578125! disk.nvme0c0n1.in‚ 153.9765625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 153.9765625 disk.rbd0.out‚0$ memory_percent‚90.86451530456543( proc.memory.availableMB‚ 1496.7578125 disk.sda2.in‚0 disk.dm-1.out‚ 26.9140625 disk.rbd1.in‚0 disk.sda1.out‚0 ¨Ów“€‚(:ˆ ƒŸæÔ§‚ÚÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VõW%‚(:š ƒŸæÔ«çšÙ% disk.nvme0n1.out‚2458.49169921875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2781.09765625 proc.cpu.threads‚17 disk.dm-0.in‚ 153.9765625 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚340929 disk.sda1.out‚0 cpu‚25.068842559635172! disk.nvme0c0n1.in‚ 153.9765625" disk.dm-0.out‚2458.49169921875 disk.sda.in‚0 disk.rbd2.out‚17.16455078125$ memory_percent‚90.86606502532959* proc.memory.percent‚16.974472999572754 disk.rbd0.out‚0 network.sent‚ 1.234009e+06) disk./.usagePercent‚74.38233907038708 disk.dm-1.in‚0' disk.nvme0c0n1.out‚2458.49169921875 disk.sda2.out‚0 disk.dm-1.out‚ 27.12890625 disk.sda.out‚ 27.12890625) proc.memory.availableMB‚ 1496.50390625 disk.sda2.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1108.3506050109863 disk.nvme0n1.in‚ 153.9765625 disk.sda3.in‚0 disk.sda3.out‚ 27.12890625ŽÍÇž‚‚(:“ ’ŸæÔÝ¿áÕ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶îá샂(:’ ’ŸæÔÝþñ× network.sent‚ 1.240267e+06 network.recv‚345097 disk.sda3.out‚ 27.64453125 disk.rbd0.out‚0 disk.rbd2.out‚17.16455078125! disk.nvme0c0n1.in‚ 153.9765625 disk.dm-0.in‚ 153.9765625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0' proc.memory.availableMB‚ 1496.984375" disk./.usageGB‚1108.3505859375 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0! disk.dm-0.out‚2458.8759765625 disk.dm-1.out‚ 27.64453125 cpu‚25.068243994485822 disk.sda.in‚0 disk.sda.out‚ 27.64453125* proc.memory.percent‚16.974472999572754) disk./.usagePercent‚74.38233779034941& disk.nvme0c0n1.out‚2458.8759765625 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2781.09765625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 153.9765625$ disk.nvme0n1.out‚2458.8759765625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0$ memory_percent‚90.86313247680664Þ-¹b0„+ _wandbruntime‚751  _runtime‚751'•½>Ç…Á ' lr-AdamW/pg2‚0.00015989603502509266 trainer/global_step‚1049 % _timestamp‚1.7884486601086762e+09 _runtime‚ 751.663116596 _step‚80 ' lr-AdamW/pg1‚0.00015989603502509266îd؈Štrainer/global_step‚1049 % _timestamp‚1.7884486601086762e+09 _runtime‚ 751.663116596 _step‚80 ' lr-AdamW/pg1‚0.00015989603502509266 ' lr-AdamW/pg2‚0.00015989603502509266Pq02Ù0‡+ _wandbruntime‚751  _runtime‚751h6!dMˆ³O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13438317 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58522310515234 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14724398 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64753929858560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460778H ½'‰·O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460778 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13438317 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58522310515234 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14724398 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 861106976768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460778 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64753929858560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14724396Qx« 0Š+ _wandbruntime‚752  _runtime‚752áE› Å&‹¿M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65722052076544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 906657700864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60105882733056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 861926417920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 _runtime‚ 752.341700168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚82 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8590268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 % _timestamp‚1.7884486607868187e+09’εP¿ ŒÃM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚82 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8590268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 % _timestamp‚1.7884486607868187e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDevic7Au eStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65722052076544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19460692 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 906657700864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60105882733056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 861926417920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 861270809088 _runtime‚ 752.341700168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4860466 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 901437187072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1099 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948868385280 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚85 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67836466741760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 898125378226 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61308352382642 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62005237215744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62907348001280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5087676 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 787.805057751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20368290 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788448696250409e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20386830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68785499517952 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62005237215744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62907348001280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5087676 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 787.805057751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20368290 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788448696250409e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20386830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68785499517952 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 901437187072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1099 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948868385280 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 902092782592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚85 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67836466741760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 YHDeviceStatsMonitor.on_train_bäB(Hatch_start/requested_bytes.small_pool.freed‚ 898125378226 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61308352382642 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20386744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60465148174336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884486969209177e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20405310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 902912223744 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35844524 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62963846869504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15439212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68847831384064 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 899596243606 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚86 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15439212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35825956 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61364072948166 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 788.47603364 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8998668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5092809 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 788.47603364 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15439214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8998668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5092809 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20386744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60465131134976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60465148174336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884486969209177e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20405310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 902912223744 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35844524 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021V…¯s0+ _wandbruntime‚788  _runtime‚788ufËtSžÍ .loss/FRt3/detection_loss‚2.575120112475227 1loss/SRqqt1/detection_loss‚2.4410078148452605 1loss/SRqqt3/detection_loss‚2.6315180235438875 .loss/FBt3/detection_loss‚4.158322722688317 /loss/FRt2/assignment_loss‚3.576619667066655 0loss/FBt2/assignment_loss‚0.3168479189094577 epoch‚0 _step‚87 1loss/SRqqt1/assignment_loss‚2.697664328983852 1loss/SRqqt4/assignment_loss‚2.756244753969127 /loss/FRt1/detection_loss‚2.7871458959288713 %loss/total_loss‚74.09632780232792 /loss/FRt4/assignment_loss‚5.439055456834681 /loss/FRt4/detection_loss‚2.4944507909171723 1loss/SRqqt4/detection_loss‚3.0255907845908196 /loss/FBt2/detection_loss‚2.2423313475446776 _runtime‚ 788.477688748 /loss/FRt1/assignment_loss‚5.213840868414902 /loss/FBt1/detection_loss‚13.222168922424316 /loss/FRt3/assignment_loss‚5.658638091648326 0loss/FBt4/assignment_loss‚0.4067085134138324 1loss/SRqqt2/detection_loss‚1.3064417544927738 % _timestamp‚1.7884486969233415e+09 -loss/FRt2/detection_loss‚1.96714855898434 2loss/SRqqt2/assignment_loss‚2.5806560291872396 2loss/SRqqt3/assignment_loss‚1.9488585434026189 0loss/FBt1/assignment_loss‚0.6145134270191193 0loss/FBt3/assignment_loss‚0.4441446234670184 trainer/global_step‚1099 /loss/FBt4/detection_loss‚3.5912888515754275½·vWŸÑ -loss/FRt2/detection_loss‚1.96714855898434 2loss/SRqqt2/assignment_loss‚2.5806560291872396 2loss/SRqqt3/assignment_loss‚1.9488585434026189 0loss/FBt1/assignment_loss‚0.6145134270191193 0loss/FBt3/assignment_loss‚0.4441446234670184 trainer/global_step‚1099 /loss/FBt4/detection_loss‚3.5912888515754275 .loss/FRt3/detection_loss‚2.575120112475227 1loss/SRqqt1/detection_loss‚2.4410078148452605 1loss/SRqqt3/detection_loss‚2.6315180235438875 .loss/FBt3/detection_loss‚4.158322722688317 /loss/FRt2/assignment_loss‚3.576619667066655 0loss/FBt2/assignment_loss‚0.3168479189094577 epoch‚0 _step‚87 1loss/SRqqt1/assignment_loss‚2.697664328983852 1loss/SRqqt4/assignment_loss‚2.756244753969127 /loss/FRt1/detection_loss‚2.7871458959288713 %loss/total_loss‚74.09632780232792 /loss/FRt4/assignment_loss‚5.439055456834681 /loss/FRt4/detection_loss‚2.4944507909171723 1loss/SRqqt4/detection_loss‚3.0255907845908196 /loss/FBt2/detection_loss‚2.2423313475446776 _runtime‚ 788.477688748 /loss/FRt1/assignment_loss‚5.213840868414902 /loss/FBt1/detection_loss‚13.222168922424316 /loss/FRt3/assignment_loss‚5.658638091648326 0loss/FBt4/assignment_loss‚0.4067085134138324 1loss/SRqqt2/detection_loss‚1.3064417544927738 % _timestamp‚1.7884486969233415e+09Wqõ0Hž ‚(:“ ¿ŸæÔ†‡ÜÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;“¡‚(:’ ¿ŸæÔ¤š£Ø! disk.nvme0c0n1.in‚ 153.9765625% disk./.usageGB‚1108.3507118225098 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 18.9775390625% disk.nvme0c0n1.out‚2654.556640625 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 proc.cpu.threads‚17 disk.sda.in‚0# memory_percent‚90.8672571182251# proc.memory.rssMB‚ 2781.09765625 network.sent‚ 1.300942e+06 network.recv‚359040) disk./.usagePercent‚74.38234623859805 disk.rbd3.out‚0 disk.sda.out‚ 29.41015625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.9765625 disk.dm-0.out‚2654.556640625 disk.sda3.out‚ 29.41015625 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.dm-1.out‚ 29.41015625) proc.memory.availableMB‚ 1496.30859375* proc.memory.percent‚16.974472999572754 cpu‚25.067219034086634# disk.nvme0n1.out‚2654.556640625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0“)Ýòž¢‚(:“ ΟæÔì›àÕ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.915799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a}¬Õ£‚(:” ΟæÔÐùì× disk.sda1.in‚0$ disk.nvme0n1.out‚2655.5166015625# proc.memory.rssMB‚ 2781.09765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 18.9775390625! disk.nvme0c0n1.in‚ 153.9765625 disk.dm-1.in‚0* proc.memory.percent‚16.974472999572754 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.9765625! disk.dm-0.out‚2655.5166015625 disk.dm-1.out‚ 29.86328125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.out‚0$ memory_percent‚90.86751937866211 network.recv‚364308 disk.rbd3.in‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚2655.5166015625 disk.sda.out‚ 29.86328125 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.out‚0% disk./.usageGB‚1108.3505935668945 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 disk.sda1.out‚0 cpu‚25.066338121398775 network.sent‚ 1.350834e+06 disk.rbd1.in‚0 disk.sda3.out‚ 29.86328125 disk.sda2.in‚0' proc.memory.availableMB‚ 1496.265625) disk./.usagePercent‚74.38233830236449 disk.rbd3.out‚0nàS0¤+ _wandbruntime‚823  _runtime‚823æ½õÄ¥¾ _runtime‚ 823.889354698 _step‚88 & lr-AdamW/pg1‚0.0001751387457043198 & lr-AdamW/pg2‚0.0001751387457043198 trainer/global_step‚1149 $ _timestamp‚1.788448732334963e+09HŠëȦ & lr-AdamW/pg1‚0.0001751387457043198 & lr-AdamW/pg2‚0.0001751387457043198 trainer/global_step‚1149 $ _timestamp‚1.788448732334963e+09 _runtime‚ 823.889354698 _step‚88XÁýè0§+ _wandbruntime‚823  _runtime‚823É;3x¨´O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9398902 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚1149 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65765312151040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63155433532416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21294342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992065942016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37438880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16125998 _runtime‚ 823.890658365 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14719323 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9398900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65765r8B¥B985749504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71911045626880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64094394250050 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 939615147842 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 943078588416 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 942422992896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 _step‚89 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 $ _timestamp‚1.788448732335895e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16125996 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65765312151040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63155433532416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21294342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992065942016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 943078588416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605Y3”Ö0ª+ _wandbruntime‚824  _runtime‚824¾ÿpÄ&«¾M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64879260199936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 824.565651186 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1149 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16140014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788448733010658e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21331362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64150786285350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _step‚90 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936?±é¬ÂM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64879260199936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 824.565651186 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1149 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16140014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788448733010658e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21331362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70980654362624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64150786285350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _step‚90 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batcúÊÄžßh_end/requested_bytes.all.peak‚ 9643691477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376ZI¢ 0­+ _wandbruntime‚824  _runtime‚824vó8¢T®Î 0loss/FBt4/assignment_loss‚0.6652125106876383 /loss/FRt2/detection_loss‚2.5549430535087048 /loss/FRt3/assignment_loss‚5.064332110541208 2loss/SRqqt2/assignment_loss‚2.5013665460251473 1loss/FBt2/assignment_loss‚0.31048875712280705 _step‚91 0loss/FBt3/assignment_loss‚0.5091355956045744 /loss/FRt4/detection_loss‚3.0591866738266416 /loss/FBt2/detection_loss‚2.2467195115304808 %loss/total_loss‚66.16665995296499 trainer/global_step‚1149 _runtime‚ 824.567454945 2loss/SRqqt3/assignment_loss‚2.3074093287189803 /loss/FBt1/assignment_loss‚0.568977708928287 0loss/SRqqt1/detection_loss‚2.491700307528178 epoch‚0 0loss/FRt2/assignment_loss‚3.0094772014977798 2loss/SRqqt4/assignment_loss‚2.6513419015066964 .loss/FRt3/detection_loss‚2.711000063589641 .loss/FBt1/detection_loss‚7.870645364125569 /loss/FBt4/detection_loss‚2.3863617604721044 1loss/SRqqt2/detection_loss‚1.5624779433679037 /loss/FRt1/detection_loss‚2.6008582539028593 /loss/FRt4/assignment_loss‚5.053518546952142 2loss/SRqqt1/assignment_loss‚2.2079732696215313 1loss/SRqqt4/detection_loss‚2.6852020578724995 .loss/FRt1/assignment_loss‚4.85521760251787 0loss/SRqqt3/detection_loss‚2.382611995562911 /loss/FBt3/detection_loss‚3.9105018879528397 $ _timestamp‚1.788448733013069e+09ë\L•X¯Ò 0loss/FBt4/assignment_loss‚0.6652125106876383 /loss/FRt2/detection_loss‚2.5549430535087048 /loss/FRt3/assignment_loss‚5.064332110541208 2loss/SRqqt2/assignment_loss‚2.5013665460251473 1loss/FBt2/assignment_loss‚0.31048875712280705 _step‚91 0loss/FBt3/assignment_loss‚0.5091355956045744 /loss/FRt4/detection_loss‚3.0591866738266416 /loss/FBt2/detection_loss‚2.2467195115304808 %loss/total_loss‚66.16665995296499 trainer/global_step‚1149 _runtime‚ 824.567454945 2loss/SRqqt3/assignment_loss‚2.3074093287189803 /loss/FBt1/assignment_loss‚0.568977708928287 0loss/SRqqt1/detection_loss‚2.491700307528178 epoch‚0 0loss/FRt2/assignment_loss‚3.0094772014977798 2loss/SRqqt4/assignment_loss‚2.6513419015066964 .loss/FRt3/detection_loss‚2.711000063589641 .loss/FBt1/detection_loss‚7.870645364125569 /loss/FBt4/detection_loss‚2.3863617604721044 1loss/SRqqt2/detection_loss‚1.5624779433679037 /loss/FRt1/detection_loss‚2.6008582539028593 /loss/FRt4/assignment_loss‚5.053518546952142 2loss/SRqqt1/assignment_loss‚2.2079732696215313 1loss/SRqqt4/detection_loss‚2.6852020578724995 .loss/FRt1/assignment_loss‚4.85521760251787 0loss/SRqqt3/detection_loss‚2.382611995562911 /loss/FBt3/detection_loss‚3.9105018879528397 $ _timestamp‚1.788448733013069e+09[.ïs’°‚(:‡ ÝŸæÔ¢ŽßÕ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óXj| ±‚(:• ÝŸæÔàêÙ disk.sda.out‚ 30.34765625% disk./.usageGB‚1108.3506698608398 disk.rbd1.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚2752.9677734375 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 disk.sda2.out‚0 disk.dm-1.out‚ 30.34765625 disk.rbd3.out‚0 disk.sda3.out‚ 30.34765625# proc.memory.rssMB‚ 2781.09765625 network.sent‚ 1.356375e+06 disk.sda2.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2752.9677734375$ memory_percent‚90.86751937866211 cpu‚25.066266250825393 network.recv‚368257 disk.drbd1777.in‚0! disk.dm-0.out‚2752.9677734375 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚19.14208984375) disk./.usagePercent‚74.38234342251516 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd3.in‚0' proc.memory.availableMB‚ 1496.265625* proc.memory.percent‚16.974472999572754 disk.nvme0n1.in‚ 153.9765625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 153.9765625¨“²‚(:ˆ ìŸæÔâ¦ãÕ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.364' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S><䳂(: ìŸæÔŠè«Ø# proc.memory.rssMB‚ 2781.09765625! disk.dm-0.out‚2753.6416015625 disk.sda2.in‚0 disk.rbd3.out‚0 network.recv‚374150 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0 disk.rbd2.out‚19.14990234375! disk.nvme0c0n1.in‚ 153.9765625* proc.memory.percent‚16.974472999572754 disk.rbd0.out‚0 disk.dm-1.out‚ 30.609375 network.sent‚ 1.406407e+06 disk.drbd1777.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚2753.6416015625 cpu‚25.06603107808123% disk./.usageGB‚1108.3506507873535 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 disk.rbd3.in‚0 disk.rbd0.in‚0$ memory_percent‚90.86811542510986 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 30.609375 disk.sda3.out‚ 30.609375) proc.memory.availableMB‚ 1496.16796875) disk./.usagePercent‚74.38234214247748$ disk.nvme0n1.out‚2753.6416015625 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17Lz,¶‘´‚(:† ûŸæÔ®—ÆÕ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{е‚(:… ûŸæÔú£™Ù disk.sda.in‚0 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1496.09765625 disk.sda3.out‚31.25 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 153.9765625) disk./.usagePercent‚74.38233855837201% disk./.usageGB‚1108.3505973815918 disk.sda.out‚31.25! disk.dm-0.out‚2753.8720703125$ memory_percent‚90.86854457855225! disk.nvme0c0n1.in‚ 153.9765625 disk.rbd1.out‚ 0.306640625 network.sent‚ 1.411625e+06 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 153.9765625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚2753.8720703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2781.09765625* proc.memory.percent‚16.974472999572754 network.recv‚378244 disk.sda1.out‚0& disk.nvme0c0n1.out‚2753.8720703125 disk.dm-1.out‚31.25 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚25.065707914959784 disk.rbd2.out‚19.29052734375“0¶+ _wandbruntime‚859  _runtime‚859 éÆ·À trainer/global_step‚1199 $ _timestamp‚1.788448768157945e+09 _runtime‚ 859.712407825 _step‚92 ' lr-AdamW/pg1‚0.00018276010104393338 ' lr-AdamW/pg2‚0.00018276010104393338þ{’Ê¸Ä trainer/global_step‚1199 $ _timestamp‚1.788448768157945e+09 _runtime‚ 859.712407825 _step‚92 ' lr-AdamW/pg1‚0.00018276010104393338 ' lr-AdamW/pg2‚0.00018276010104393338\_ü"«0¹+ _wandbruntime‚859  _runtime‚859=qËÀ½'º·O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68623949899264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983408798720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9807300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65900657099776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68624623497728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67640541100544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74001540508160 trainer/global_step‚1199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39047190 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67640559103488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5552497 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16826798 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67640541100544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66880436117458 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 980450487250 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39065732 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15359799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22220394 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884487681587002e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74001540508160 trainer/global_step‚1199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39047190 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67640559103488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5552497 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16826798 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67640541100544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1035310102528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66880436117458 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 980450487250 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39065732 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15359799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22220394 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884487681587002e+09 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 _step‚94 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚1199 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes^q\ƒŸ.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67696894139392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 % _timestamp‚1.7884487688312893e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344Ô~•Ë&¾ÅM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 _step‚94 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚1199 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67696894139392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 % _timestamp‚1.7884487688312893e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414^0ú^$0¿+ _wandbruntime‚860  _runtime‚860î“·©TÀÎ 0loss/FBt1/assignment_loss‚0.6240808931179345 1loss/SRqqt4/detection_loss‚3.4269763568173284 .loss/FBt1/detection_loss‚8.750631093978882 2loss/SRqqt3/assignment_loss‚2.1096806195942133 1loss/SRqqt4/assignment_loss‚2.826086339743241 1loss/FBt2/assignment_loss‚0.30933201230506585 .loss/FRt4/detection_loss‚2.828043960426983 1loss/SRqqt2/detection_loss‚1.2667582836742937 /loss/SRqqt1/assignment_loss‚2.0533076296019 2loss/SRqqt2/assignment_loss‚2.3576367853040043 /loss/FRt3/detection_loss‚2.4860756650115503 1loss/SRqqt3/detection_loss‚2.8339736461639404 .loss/FBt3/detection_loss‚4.618509647995234 epoch‚0 % _timestamp‚1.7884487688332958e+09 1loss/FBt3/assignment_loss‚0.44083753841696305 _runtime‚ 860.387622176 0loss/FRt2/assignment_loss‚3.0084342285990715 /loss/FRt3/assignment_loss‚4.562284668286641 0loss/FBt4/assignment_loss‚0.5145746430819693 /loss/FRt2/detection_loss‚2.4247632904776504 1loss/SRqqt1/detection_loss‚2.2652726499147193 _step‚95 /loss/FRt1/assignment_loss‚3.946082143073386 .loss/FRt1/detection_loss‚2.429110868814144 %loss/total_loss‚65.32628832434358 /loss/FBt2/detection_loss‚1.8190869339579219 /loss/FBt4/detection_loss‚1.8858312609330232 trainer/global_step‚1199 /loss/FRt4/assignment_loss‚5.538917165053518©œ¿XÁÒ 2loss/SRqqt3/assignment_loss‚2.1096806195942133 1loss/SRqqt4/assignment_loss‚2.826086339743241 1loss/FBt2/assignment_loss‚0.30933201230506585 .loss/FRt4/detection_loss‚2.828043960426983 1loss/SRqqt2/detection_loss‚1.2667582836742937 /loss/SRqqt1/assignment_loss‚2.0533076296019 2loss/SRqqt2/assignment_loss‚2.3576367853040043 /loss/FRt3/detection_loss‚2.4860756650115503 1loss/SRqqt3/detection_loss‚2.8339736461639404 .loss/FBt3/detection_loss‚4.618509647995234 epoch‚0 % _timestamp‚1.7884487688332958e+09 1loss/FBt3/assignment_loss‚0.44083753841696305 _runtime‚ 860.387622176 0loss/FRt2/assignment_loss‚3.0084342285990715 /loss/FRt3/assignment_loss‚4.562284668286641 0loss/FBt4/assignment_loss‚0.5145746430819693 /loss/FRt2/detection_loss‚2.4247632904776504 1loss/SRqqt1/detection_loss‚2.2652726499147193 _step‚95 /loss/FRt1/assignment_loss‚3.946082143073386 .loss/FRt1/detection_loss‚2.429110868814144 %loss/total_loss‚65.32628832434358 /loss/FBt2/detection_loss‚1.8190869339579219 /loss/FBt4/detection_loss‚1.8858312609330232 trainer/global_step‚1199 /loss/FRt4/assignment_loss‚5.538917165053518 0loss/FBt1/assignment_loss‚0.6240808931179345 1loss/SRqqt4/detection_loss‚3.4269763568173284 .loss/FBt1/detection_loss‚8.750631093978882_hûìH‘‚(:† Š æÔ•¢ØÕ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T A?Â(: Š æÔÝß³Ø disk.dm-1.out‚ 31.73046875 disk.sda.out‚ 31.73046875 disk.rbd2.out‚20.86474609375 disk.nvme0n1.in‚ 153.9765625 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 153.9765625# disk.nvme0n1.out‚2852.111328125# proc.memory.rssMB‚ 2856.86328125% disk./.usageGB‚1108.3510856628418 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚382679 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 31.73046875( disk./.usagePercent‚74.3823713273364 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd2.in‚0% disk.nvme0c0n1.out‚2852.111328125 disk.drbd1777.in‚0( proc.memory.availableMB‚ 1420.0234375 disk.dm-0.in‚ 153.9765625 disk.dm-0.out‚2852.111328125$ memory_percent‚91.33286476135254* proc.memory.percent‚17.436909675598145 network.sent‚ 1.41788e+06 disk.rbd0.in‚0 cpu‚25.08419212537614 proc.cpu.threads‚17 disk.sda2.in‚0h6‹­“Ä‚(:ˆ ™ æÔÜèäÕ gpu.0.gpu‚42.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰A~ÛÅ‚(:‡ ™ æÔÄöœÛ disk.rbd0.out‚0( proc.memory.availableMB‚ 1412.5546875# proc.memory.rssMB‚ 2864.47265625 disk.sda.in‚0 disk.sda.out‚32.34375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.48335361480713 disk.dm-0.out‚2852.376953125 disk.dm-1.out‚32.34375 disk.drbd1777.in‚0$ memory_percent‚91.37845039367676 proc.cpu.threads‚17 disk.rbd1.out‚ 0.306640625% disk.nvme0c0n1.out‚2852.376953125# disk.nvme0n1.out‚2852.376953125 cpu‚25.167471869655603 network.recv‚387758 disk.sda2.out‚0 disk.rbd2.out‚20.95849609375 disk.nvme0n1.in‚ 153.9765625 disk.dm-0.in‚ 153.9765625 disk.rbd3.in‚0 disk.sda1.in‚0 network.sent‚ 1.467026e+06# disk./.usageGB‚1108.35103225708 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.out‚32.34375! disk.nvme0c0n1.in‚ 153.9765625 disk.drbd1777.out‚0) disk./.usagePercent‚74.38236774323092 disk.sda2.in‚0¯ár;Æ‚(:’ ¨ æÔÝ­¾Õ gpu.0.gpu‚57.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.874399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áwr½Ç‚(:” ¨ æÔÞ¢†Ø" disk.dm-0.out‚2950.47900390625 disk.rbd0.in‚0* proc.memory.percent‚17.441725730895996% disk./.usageGB‚1108.3511772155762 disk.sda2.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2950.47900390625 disk.nvme0n1.in‚ 153.9765625% disk.nvme0n1.out‚2950.47900390625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 network.sent‚ 1.473263e+06 disk.dm-0.in‚ 153.9765625 disk.sda.out‚ 33.0390625 disk.sda3.out‚ 33.0390625$ memory_percent‚91.33892059326172 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd2.out‚20.95849609375! disk.nvme0c0n1.in‚ 153.9765625 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 cpu‚25.254066272959335 network.recv‚392372) disk./.usagePercent‚74.38237747151722 disk.dm-1.out‚ 33.0390625 disk.rbd1.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 1419.03125# proc.memory.rssMB‚ 2857.65234375•0úÛÈ‚(:’ · æÔñ…ÕÕ gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.874399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áDFòÉ‚(: · æÔË÷Ø) proc.memory.availableMB‚ 1419.01953125! disk.dm-0.out‚2950.8115234375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 21.123046875 disk.dm-0.in‚ 153.9765625 disk.dm-1.out‚ 33.5234375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 33.5234375 cpu‚25.333199341462347) disk./.usagePercent‚74.38237644748708 disk.sda2.out‚0 disk.sda.out‚ 33.5234375 disk.sda3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2857.76953125& disk.nvme0c0n1.out‚2950.8115234375 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.in‚0 disk.rbd0.in‚0 network.sent‚ 1.478429e+06 disk.rbd2.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚1108.351161956787$ disk.nvme0n1.out‚2950.8115234375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.out‚0$ memory_percent‚91.33899211883545( proc.memory.percent‚17.4424409866333 network.recv‚396211! disk.nvme0c0n1.in‚ 153.9765625 proc.cpu.threads‚17 disk.sda.in‚0¥ÃxdÊb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP¢ÏÛmÝbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPAÒ¯ÅvÞbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP'Ò‚:tßbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP*‘2pàbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JP6¥‡yábt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÜ%çuâbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP€m/yãbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPgÌû)väbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP™pÍ‚våbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÏU–~æby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPUNiçbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPÊ‹Ç+tèbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP’ÊL_sébn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPÙcyêbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP-ÿxqëbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPýÜ>‹oìbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPÿÔ?@tíbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP‚Bw"qîbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPáÚlïbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JP2—»ìlðbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JPû$‡Údñb_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP1â$r÷bm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP᪦løbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP‹\)•mùbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPÿ„iúbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPœñîÇtûbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP×õEyübt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÔÿà´oýbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPë‡sþbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP+Iiÿbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JP4Êo€bj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPçÕŸwbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÔÂO´x‚bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP“Æ­nƒbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JP'½)m„bh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP¥¤ì"-…bg DDeviceStatsMonitor.on_validation_batchƒ^ ?_start/inactive_split.all.peak"trainer/global_step2:JPiï‡!k†bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPoðÞzv‡bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPýctyuˆbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPÆ2°mt‰bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPïÚkŠbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPãpöÔq‹bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPØQeŒb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPÅù­Jgbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPùÄÓrŽbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP¼§ë-vbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP"ùZlbg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP_9Ût‘bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPÓ㵞m’bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP@„¡št“bo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP¢Bçv”bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP^Þ€?t•bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP4€ùes–bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPÛÀ÷Lq—bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPÓ`RPm˜bh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JP*¥ãKo™bj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPɯzUnšbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP:Ѫîl›bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPÃù4Ówœbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPggj¡~by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP3®îvmžbh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPøµ?îxŸbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÚ?îp bk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JPB,k¡bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPÝ\^­h¢bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPYØ–Ãz£bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP,8²Gy¤bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP&"is¥bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPrèݱx¦bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPï¡Tœm§bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP×gj¨be BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPw¢Nw©br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP 'oªbj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JPª5äÇz«bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPE;§Cp¬bk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPÃIŽ‹o­bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JP~ îr®bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP4ꎸy¯bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPlƒÉðk°bf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPþÛžt±bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP¸:˜jn²bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP²^´Su³bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP»6ÆÇt´bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPGßÃírµbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPñINlr¶bm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP“ò¸Mp·bk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPbÇ9@p¸bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPÈ1!t¹bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPÿeÚ#jºbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPHyø^u»bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPxe‡u¼bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPó Kµr½bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPK w¾br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP¸Ðím¿bh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP‹'¢sÀbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP™ÕatÁbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPõ(:‡gÂbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JP[.é6qÃbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JP‰B£GlÄbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPFèÙ0Å+ _wandbruntime‚915  _runtime‚915LÄê¾+*Æ¥T MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚40262510 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507950476 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚70008760903168 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510430720 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1010164851200 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1009672423424 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚76473958826496 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1006447712518 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚69185540332806 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _runtime‚ 915.429546768 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚77535980766208 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚96 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚154 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22766060 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492427776 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17496452 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚22781251 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510430720 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492427776 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062021939712 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643691477 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚17496452 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22630 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 $ _timestamp‚1.788448823874417e+09 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚76473934886400 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9954819072 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1061910387712 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚22781251 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71018925754368 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚70008742900224 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚70008760903168 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71018415323648 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863048704 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚68179075580928 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861845973 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1009672423424 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚71018925754368 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22766060 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚70008742900224 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 135492096 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚10207676 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15193 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚70008760903168 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510430720 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1010164851200 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1009672423424 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚68179092620288 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚76473958826496 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1006447712518 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚69185540332806 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _runtime‚ 915.429546768 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9954819072 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚77535980766208 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚96 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16039405 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2`·ŠÊiÈbd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPÓ›ÍjÉbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPýz,…nÊbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JP«•‡vËbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÉû!›jÌbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPØL[oÍbj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPB_¨åcÎb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP„ðÜ3gÏbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP5d, tÐbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP}ØVAqÑbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP·²¥3nÒbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP„í<ÎoÓbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP?nÔbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP­2ÚskÕbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPfKCnÖbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPugß*l×bg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPI‹wsØbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÚíéuÙbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPÁ+„sÚbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPL H5|Ûbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP„º)¶uÜbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP‰_EFmÝbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP4$²ÀhÞbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPs!ïußbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP1Ëâ®qàbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP1GfMpábk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPGÛeâb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JP¦Çr=kãbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPÇk|5käbf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPóõ?1åbu RDeviceStatsMonitor.on_validation_batch_end¾u I/inactive_split_bytes.large_pool.current"trainer/global_step2:JP›u+;wæbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPº6ƒímçbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPH*’>rèbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP>V'oébj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPšäL¶têbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPÿóÄbëb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JP?¿ßŽeìb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP|ͱjíbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPh>Êxîbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPóH‰Loïbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JP1dòŽtðbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPfÍØ•tñbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP’Çbòb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP…§Ékóbf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPI tôbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP3eƒÏrõbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPX‡~löbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JP’‚n.m÷bh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JPóo4‡iøbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPOë;Kpùbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÉ“wúbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP­Öúörûbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP­\|™nübi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JP‹…5[výbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPƒm­Õkþbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPD´¤clÿbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPµÊ§Ým€bh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP‰£Å5tbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPã”×£s‚bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPhÚê°fƒba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP¬Qõçl„bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPfj…be BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPAn·r†bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP¤ÒFm‡bh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP†,txˆbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP·˜Ìc‰b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JP¶frgŠbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPw¡3Nk‹bf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP€‘umŒbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPõßjbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP3X³"vŽbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPˆ´“¤pbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPøÞÕàjbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPã$¦i‘bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP,Ø(™u’bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP¸Ïyi“bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPv¸"wr”bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPõ;áìq•bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÓBÚq–bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPó¹°r—bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPSÆŽr˜bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP}Ûaw™br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP?,tÀjšbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP°;¤p›bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP¢¸Wrœbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPfKfba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPiÃqžbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP«xblŸbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP¢ú©r bm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPÕ(šm¡bh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JP­ur¢bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPýsìw£br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP„¿os¤bn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP·p^¯z¥bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP$Õg¦bb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPÛ@ZÕp§bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPó¨m´p¨bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPÓg |©bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP~t-hªbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPǾ·¶c«b^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP¶5„¦n¬bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPØ·­·k­bf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP$YQcw®br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPXfâr¯bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP´ÊÑv°bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPV vÿk±bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP[õ:a²b\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPQ{qo³bj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP¶RΚq´bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPj o¨oµbj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPˆªÖm¶bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPfßÎéo·bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP ¯0w¸br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPò›<=d¹b_ )¢ó6)Ä°R [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490911116 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9954819072 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚77574086134272 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚69221197701702 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510430720 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚16050559 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1062275514368 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9954819072 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚70045572929024 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1010529977856 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚17507258 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22773362 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71056120909824 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1010529977856 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1010037550080 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _step‚97 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚5836379 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861845973 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1208 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚17507256 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 _runtime‚ 916.343609758 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚22788553 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚70045590931968 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135492096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15191 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062387066368 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1006811321926 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚10214182 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884488247887504e+09 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _step‚97 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚5836379 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861845973 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1208 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚17507256 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 _runtime‚ 916.343609758 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚22788553 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚70045590931968 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135492096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15191 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062387066368 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1006811321926 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚10214182 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884488247887504e+09 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 BJûbE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬‰WJübE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÕk9×JýbE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÿX HþbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP> sNÿbI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP–ã¾ðN€ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP:ô0N bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÅûòN‚ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPg’šJƒ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPi¡^^N„ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPv&jJ… bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPFü•”N† bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP @¥N‡ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP‚ Jˆ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP@‰ÍH‰ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿FC JŠ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPâ…ŸH‹ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP˜ ÜšNŒ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¦_üñN bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPzJ«ÞHŽ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP|üºûH bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPe«­J bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPà{ï_N‘ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPuN’ bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP&õÇN“ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPc-ˆEH” bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPfXûíN• bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPºpþ²;– b6 jet/accuracy_1_of_5"trainer/global_step2:JPQ:p¯@— b; particle/accuracy_2_of_5"trainer/global_step2:JPÙØ¿2J˜ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP;Wé%J™ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPc÷8Jš bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP D2‰H› bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPýÏ·Jœ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP³»O'J bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP…ìNž bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPe™qVNŸ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP5[ H  bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP K¿@¡ b; particle/accuracy_4_of_7"trainer/global_step2:JPéÑ/N¢ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP~\_ N£ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPHœWÕN¤ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPRbÊH¥ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÝ4×{J¦ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPN‘Ò£H§ bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP2h:J¨ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPB‰sH© bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¯ØåNª bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPzNrJ« bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP->õ©H¬ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP8†=*H­ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPeCœñJ® bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÖ•çN¯ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPNñAëN° bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP¹¨BTJ± bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP½Ÿl%H² bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPê¤qJ³ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPQ÷N´ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPš`ŸuJµ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPïÕê~N¶ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP4÷jJ· bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPz¸ 'N¸ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPVýQiN¹ bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPf­] Nº bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPIÔ°;» b6 jet/accuracy_1_of_6"trainer/global_step2:JPhíÉJ¼ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÁ;ú4H½ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPN±~ÉH¾ bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP–êçìH¿ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÑ’· NÀ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPº¯Y@Á b; particle/accuracy_1_of_2"trainer/global_step2:JPÕSc(J bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPZåAõHà bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP¿¶ŠËNÄ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÌé>dJÅ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPé¨WJNÆ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPTŒòzJÇ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‘~JÈ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPg ĺHÉ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÐ =HÊ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP-„€tHË bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÆ÷ÀHÌ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP4´ö/;Í b6 jet/accuracy_2_of_3"trainer/global_step2:JPpS#bJÎ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPäYaHÏ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP_õGHÐ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPP1 ‹NÑ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÃ!ß<Ò b7 particle/sensitivity"trainer/global_step2:JP gHÓ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPв²/HÔ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPùb@HÕ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP¹T9àJÖ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPå$×@J× bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP&a—@Ø b; particle/accuracy_3_of_3"trainer/global_step2:JPrtbHÙ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPp.æ"JÚ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPä²Ù±JÛ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÉqϹNÜ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP ‚9HÝ bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPê/=æJÞ bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP „ÃHß bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP{”ÉQHà bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP1…„¯Já bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPzÿå¾Hâ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÌÄ»¹Nã bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPJÃHä bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP='Jå bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÔúÛšJæ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP·˜(fJç bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP[JþJè bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP)&Né bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP VŒœHê bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP!¥ÖúJë bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPTŒJì bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¢u!>Jí bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPN)®,Jî bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP;Ú.Nï bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP£ðýJð bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPzS©Jñ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¢®ïHò bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP«ö#¦Hó bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP=<FHô bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP­*mÈHõ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPcd(êNö bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPmÌMJ÷ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPQ7Nø bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÆòNù bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP2þá9@ú b; particle/accuracy_6_of_6"trainer/global_step2:JPgøKJû bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPõ€@²Jü bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPï×xNý bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP3˵ENþ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP­ 2/Nÿ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP~ž:H€ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP˜,¢J bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPñ©Û:H‚ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕ9ÓNƒ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP„駭J„ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPál©ºJ… bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÍkÒN† bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP#šrnH‡ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPR«ªNˆ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPwu@‰ b; particle/accuracy_2_of_6"trainer/global_step2:JPt@¾±HŠ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP;ŸUÆJ‹ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP5—ÓDJŒ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP[?ƒN bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÔšWlJŽ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÙÎî³J bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP,ÄPgH bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP)å/p@‘ b; particle/accuracy_7_of_7"trainer/global_step2:JP},”½J’ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPå*”ŠJ“ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPbàºH” bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPƒí"N• bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP#v5N– bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP6EÞH— bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPpãBJ˜ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP”‰§[J™ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÝ!jçHš bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP žÒqN› bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPœÕÙJœ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP…=ÖÊJ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP.‰àHž bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPëeJŸ bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPë¼MbN  bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPî¾ö:N¡ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP*¯ûN¢ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP´»ÓuJ£ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP›öuÝJ¤ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP=}EH¥ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPyoõ†H¦ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPö;êN§ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÁ lJ¨ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP"s¶J© bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPFoÀãJª bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP­Â©J« bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPS™sãJ¬ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP„ ÀH­ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP@T(N® bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPzè5N¯ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP½Ô7:N° bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPú€[;J± bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPp€®ÀN² bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP„ÎJ³ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP-À.vH´ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP¹DÿíJµ bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPîz}AH¶ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄEÝH· bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÅ.+¼J¸ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPž6xJ¹ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]ŽMHº bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPÁ¤ëJ» bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP9Ÿ;¼ b6 jet/accuracy_3_of_7"trainer/global_step2:JP4SÓÓH½ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPUàÁN¾ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP~)coJ¿ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP á—DNÀ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP6áüNÁ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPþªÍJ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP°Þ5×Jà bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPûJ~WNÄ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÍ¥ÆJÅ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPbKNÆ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÑ+LJÇ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP°7ÛJÈ bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ln NÉ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP åJÊ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP^+ƯJË bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP_‘uµJÌ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP=·&ÊJÍ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPrû HÎ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP‘=›‘NÏ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¸xÇDNÐ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPi§&ÊJÑ bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPu`Ÿ¼HÒ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPU3ÄNÓ bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPJÜy¦HÔ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPúˆôïHÕ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÊ0ÑJÖ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP\['ÍN× bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP^Ñ*—JØ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPø±èHÙ bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP=³JÚ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP µä»HÛ bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPóÁC•HÜ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP´Ø<#NÝ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPv\œ‘JÞ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÝæß*Hß bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP&¿–Nà bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP¡ülCNá bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPP÷XÀ6â bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainerýG9}/global_step2:JP—­Hã bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP»¹âJä bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPb ù7@å b; particle/accuracy_1_of_3"trainer/global_step2:JP¼&†_Jæ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPˆ°Þ\Hç bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPG•èìJè bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPµÐã“Né bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPaîªHê bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPëu¹¦@ë b; particle/accuracy_2_of_7"trainer/global_step2:JPkxâ®Hì bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPp°ìWHí bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP `ŠpNî bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¯ÝCÀJï bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP°é~Jð bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPí&îJñ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP-VHò bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP8ÇžHó bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¾º€Hô bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPáí×Hõ bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPcᬳJö bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÿà¡£N÷ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP¼ øNø bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPû)’´;ù b6 jet/accuracy_2_of_5"trainer/global_step2:JPë<(È@ú b; particle/accuracy_4_of_5"trainer/global_step2:JPý·ÄÇ;û b6 jet/accuracy_7_of_7"trainer/global_step2:JPU•µJü bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP®îgJý bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPD™ãŠJþ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPD(ÁJÿ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶ïÅN€ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPDr:N bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPGÆ1N‚ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP}¼Ó Hƒ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPšÛ÷ J„ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPPÐlH¥ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÿ†WþH¦ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPéÌJÎH§ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP$w‘J¨ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP­NÀH;© b6 jet/accuracy_3_of_3"trainer/global_step2:JPŠç¬ÑGª bB validation_average_jet_accuracy"trainer/global_step2:JP@"ìáH« bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPáÃí"H¬ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÍ„UN­ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPl’ …J® bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP’“WuJ¯ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPOX»ƒ;° b6 jet/accuracy_4_of_4"trainer/global_step2:JP'9ã©J± bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPšÍ«o@² b; particle/accuracy_6_of_7"trainer/global_step2:JPõÀ†úN³ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPíV¨H´ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPoÙ ³Hµ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPiyøN¶ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP3B¹N· bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP#¾÷íJ¸ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP‘ ªÙ@¹ b; particle/accuracy_3_of_7"trainer/global_step2:JPy¡ïkJº bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP” oÞJ» bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPr¨üN¼ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP¼H8&;½ b6 jet/accuracy_4_of_5"trainer/global_step2:JPÇ ¼ÿN¾ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÜtrN¿ bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPZéqHÀ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÊ@Á b; particle/accuracy_4_of_6"trainer/global_step2:JP](ØJ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP?&3Hà bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPvÍ»ÀNÄ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP KÓ°HÅ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP¤\–HÆ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP\j,“HÇ bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP’:úONÈ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPzø{ÑHÉ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP6ËíÁNÊ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP4Ç_NË bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPOÂZJÌ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPœ²cHÍ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP<#’žNÎ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPIuùJÏ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPòÉŒáJÐ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP i_ÉNÑ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPöÊHÒ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPFIGAHÓ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP¨K0õHÔ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP*ÏûçJÕ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPˆÌYjHÖ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPËÂÏJ× bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPЀ CHØ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPð)’óHÙ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP!¿§þNÚ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPWŠ”(;Û b6 jet/accuracy_2_of_2"trainer/global_step2:JPäŸnP;Ü b6 jet/accuracy_6_of_7"trainer/global_step2:JPºvÿJÝ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPáwiHÞ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPؘ¦xNß bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÙ¨§ßJà bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¸”pHá bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPˆ½ëHâ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPÀ QUHã bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP_,ÉDNä bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP©±Ÿ_Nå bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP³âõã;æ b6 jet/accuracy_1_of_2"trainer/global_step2:JPQ ó@ç b; particle/accuracy_2_of_3"trainer/global_step2:JP ÒxÂ@è b; particle/accuracy_3_of_4"trainer/global_step2:JP”Wý¢Jé bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPo™ÖjJê bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¡ÂØ Në bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÓ-XŸNì bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPë~Ní bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¿xg`Jî bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¬Jï bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÆsNð bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPƒ¦”…Jñ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP”ÛGHò bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPy÷KŒNó bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP&šº¦Nô bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPúÏNõ bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPb‘yŠ;ö b6 jet/accuracy_5_of_5"trainer/global_step2:JPÔ)ñ@÷ b; particle/accuracy_2_of_4"trainer/global_step2:JP?þÉCHø bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP±y,Jù bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP3%ÛµNú bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP™iùšJû bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP6d,Hü bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPŠ I@ý b; particle/accuracy_3_of_6"trainer/global_step2:JPD(?Hþ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP^u] Nÿ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP_8X¾J€ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÆÄ4H bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPóS&ÁJ‚ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP^¹—Hƒ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPdAŒN„ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP!FÇN… bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP?}˜JN† bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP8lÝ]J‡ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP•£ÍŠHˆ bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP“ܯH‰ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP"ApHŠ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP‡ ɃN‹ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPeëÌHŒ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPDhKJJ bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPF×JŽ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¡ZKÄ; b6 loss/val_total_loss"trainer/global_step2:JP âk½J bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP§/¤J‘ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÐŽN’ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPoµßH“ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¯ˆ]&J” bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¶¯^5J• bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPXzMH– bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¤t­H— bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP"1üN˜ bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¸BqvJ™ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPqÕJš bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP|4J› bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPSy¸¦Jœ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP P²ÓH bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPŠU-eJž bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP׭ʼnJŸ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP[îN  bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPZZLWN¡ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPBkÞŠJ¢ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP«³U;£ b6 jet/accuracy_2_of_6"trainer/global_step2:JP8ýJ¤ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPpìÓFJ¥ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP•H 0N¦ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP²”9VN§ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP†-[N¨ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP†Ò¦J© bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÍà¬Jª bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP Òþ=N« bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPSée×J¬ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP€X½J­ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÿòŒ»H® bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPe†ä±N¯ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPó×KN° bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¿C).H± bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPwwÌH² bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPé|ÜTJ³ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP{’oH´ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP˜ä*ÉJµ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPw^ÐH¶ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPö;Š;J· bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¥\®;N¸ bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP=?‚TH¹ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP74nðJº bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP1ÀÅ&H» bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP±ø2uN¼ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP$j|íN½ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP(e»ÑN¾ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP–bN¿ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPTÆ ðJÀ bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPÿi4ôHÁ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP5ͬ;N bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPDÏX?Nà bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¯w†HÄ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÜG^–;Å b6 jet/accuracy_3_of_6"trainer/global_step2:JPÒ 3JÆ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP» NÇ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP èø HÈ bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP˜IüÉNÉ bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPo¡@ NÊ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPŠ ÙE@Ë b; particle/accuracy_1_of_6"trainer/global_step2:JPæHÌ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP NkYHÍ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP3R/@NÎ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÎÙ˜?HÏ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÀRæ€NÐ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP”‰Yø@Ñ b; particle/accuracy_1_of_1"trainer/global_step2:JPÍZÐz@Ò b; particle/accuracy_5_of_5"trainer/global_step2:JPö³¬ø;Ó b6 jet/accuracy_5_of_6"trainer/global_step2:JP\A¤ÝHÔ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPCo‹rHÕ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPšTKNÖ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPßVšÔH× bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPúNHùJØ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPçž”ºJÙ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP$QÑJÚ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP.yœNÛ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPrrùò;Ü b6 jet/accuracy_1_of_1"trainer/global_step2:JP^<ä@Ý b; particle/accuracy_1_of_5"trainer/global_step2:JPýòͲHÞ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPq?`äHß bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPçÆ’Hà bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¼ôtâJá bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPREQ:;â b6 jet/accuracy_3_of_5"trainer/global_step2:JPú×å’Jã bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPö×4ÕNä bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP]apJå bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPßRœ´Jæ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPܽ!ÍHç bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPW˜EQNè bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPEÊfµJé bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPLãÌHê bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPΖ&1Jë bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP'øì+Nì bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¡ÆŸHí bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPv ËúJî bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÖ-U÷Nï bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPþ[\ýNð bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPC¾_'Hñ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP[áÒJò bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÑ•Nó bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP³>ìJô bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÌÇüi@õ b; particle/accuracy_5_of_6"trainer/global_step2:JPܺólJö bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPãGÅH÷ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPäÙœÅJø bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³Ìç¹Jù bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP{öfJú bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÉ*Õ-Hû bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPç5Nü bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPð€lHý bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP„gDtHþ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP—K?ÄJÿ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPü1cIJ€ bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPx ô·J bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP$Ð9ÝN‚ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÏ~Nƒ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPd{J„ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÏb€N… bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPItÁKN† bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP”ƒUbH‡ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP9Z Hˆ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP8‹;J‰ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPëï‡NŠ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÐ&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 jet/accuracy_6_of_6‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2090158760547638 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14400823414325714 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9773240685462952 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.16974452137947083 ,jet/accuracy_3_of_3‚0.009620742872357368 7validation_average_jet_accuracy‚0.30357128381729126 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -jet/accuracy_4_of_4‚0.0019841270986944437 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_7‚1 trainer/global_step‚1208 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9872013330459595 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.257675439119339 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07892093807458878 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1476937085390091 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_4_of_5‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 particle/accuracy_4_of_6‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06240253150463104 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1524595469236374 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.985213041305542 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.20104163885116577 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1855064034461975 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14264264702796936 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.004576320759952068 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16835732758045197 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1332375556230545 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17428192496299744 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10802468657493591 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_2‚0.0673392191529274 jet/accuracy_6_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2071160078048706 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06680991500616074 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.44736841320991516 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.22577518224716187 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5587201118469238 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.293402761220932 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22750675678253174 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3147244155406952 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12554241716861725 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.20541724562644958 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20807723701000214 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17122112214565277 8 Purity/2FRt2SRqqt0FBt/FRt_puá|rÃÓrrity‚0.14799068868160248 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015325784683228 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.20000000298023224 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 )loss/val_total_loss‚62.96715354045511 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.32959669828414917 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.31992319226264954 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.1818181872367859 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.23275862634181976 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _step‚98 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16491220891475677 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.039509326219558716 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0024719280190765858 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.32692307233810425 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219857305288315 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02338709682226181 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014070585370063782 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.17732080817222595 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16461779177188873 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848107099533081 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3387663960456848 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944029450416565 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_5_of_5‚1 jet/accuracy_5_of_6‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 epoch‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855214953422546 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986714243888855 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23211154341697693 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 *jet/accuracy_1_of_1‚0.3153397738933563 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_5‚0.043189965188503265 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.19035471975803375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2996375262737274 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.20045220851898193 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2631579041481018 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.15207640826702118 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790908694267273 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34878140687942505 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995590448379517 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20218820869922638 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.19135800004005432 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15031619369983673 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990174412727356 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.055854372680187225 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3345472812652588 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.14888009428977966 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.15152506530284882 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11703167110681534 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26796308159828186 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992784857749939 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3015872836112976 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9390590190887451 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20092596113681793 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992784857749939 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02917531318962574 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1372319608926773 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06680991500616074 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26634088158607483 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890487790107727 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2291666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06725849211215973 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2957717776298523 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.16991108655929565 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 (particle/f_score‚0.32148921489715576 particle/accuracy_5_of_7‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.20950856804847717 particle/accuracy_4_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005094191525131464 *jet/accuracy_1_of_3‚0.6780730485916138 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2673610746860504 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9867419600486755 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003438554471358657 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3210911750793457 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.2503676414489746 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3282513916492462 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.24285714328289032 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.21913579106330872 *jet/accuracy_1_of_4‚0.7543676495552063 jet/accuracy_4_of_6‚0 ,"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.375 jet/accuracy_2_of_7‚0 _runtime‚ 929.036493865 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953367710113525 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.29436811804771423 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 (particle/accuracy‚0.8322121500968933 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04723523557186127 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.165795236825943 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.007841628976166248 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07555555552244186 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 % _timestamp‚1.7884488374795945e+09 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2933552861213684 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03707021474838257 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07533468306064606 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783734083175659 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.34377211332321167 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.18143737316131592 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.13304181396961212 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18069137632846832 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.36513930559158325 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.15357822179794312 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1988505721092224 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16148647665977478 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333134651184 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.28514930605888367 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.20218989253044128 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04140106961131096 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11384902894496918 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15665240585803986 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08580414205789566 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_5‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856102466583252 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1719929277896881 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.32652223110198975 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.163471981883049 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15747541189193726 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1580454260110855 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.14935436844825745 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2269679605960846 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0007936508045531809 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.27314817905426025 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16982144117355347 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12386590242385864 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9686248898506165 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_5‚0.8306452035903931 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9910179972648621 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10202070325613022 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5909090638160706 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12548790872097015 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6418279409408569 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9826440811157227 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156439065933228 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1522739827632904 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14646464586257935 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2122761607170105 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.8141025900840759 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9891345500946045 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_1_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814751148223877 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18086616694927216 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20321635901927948 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17348484694957733 +jet/accuracy_2_of_3‚0.15361125767230988 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.23999999463558197 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9741379022598267 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *particle/sensitivity‚0.254747211933136 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01785714365541935 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.12222222238779068 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32184848189353943 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.16316820681095123 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9649801254272461 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003968254197388887 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914575815200806 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1297452300786972 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6174558401107788 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14043210446834564 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9200000166893005 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08737215399742126 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.33309462666511536 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.19197535514831543 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 particle/accuracy_7_of_7‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.011134674772620201 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9853727221488953 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.36723792552948 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009164776653051376 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26205357909202576 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953324198722839 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2856321632862091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1275210678577423 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1441797912120819 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08847090601921082 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9874438643455505 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.2669812738895416 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.21113678812980652 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9667370319366455 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07589513808488846 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17087262868881226 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1522645503282547 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22438545525074005 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.1700061708688736 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.16974452137947083 ) Purity/1FRt*SRqqt3FBt/FRt_purity‚0.25 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14925448596477509 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2928960919380188 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21964088082313538 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14975124597549438 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19856323301792145 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16594333946704865 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548,n¹Â — ¥ 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04723523557186127 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.165795236825943 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.007841628976166248 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07555555552244186 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 % _timestamp‚1.7884488374795945e+09 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2933552861213684 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.03707021474838257 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07533468306064606 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783734083175659 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.34377211332321167 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.18143737316131592 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.13304181396961212 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18069137632846832 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.36513930559158325 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.15357822179794312 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1988505721092224 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16148647665977478 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333134651184 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ) Purity/1FRt1SRqqtRºxœù3FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.28514930605888367 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.20218989253044128 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04140106961131096 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11384902894496918 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15665240585803986 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08580414205789566 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_5‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856102466583252 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1719929277896881 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.32652223110198975 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.163471981883049 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15747541189193726 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1580454260110855 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.14935436844825745 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2269679605960846 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0007936508045531809 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.27314817905426025 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16982144117355347 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12386590242385864 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9686248898506165 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_5‚0.8306452035903931 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9910179972648621 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10202070325613022 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5909090638160706 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12548790872097015 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6418279409408569 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9826440811157227 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156439065933228 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1522739827632904 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14646464586257935 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2122761607170105 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.8141025900840759 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9891345500946045 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_1_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814751148223877 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18086616694927216 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20321635901927948 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17348484694957733 +jet/accuracy_2_of_3‚0.15361125767230988 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.23999999463558197 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9741379022598267 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *particle/sensitivity‚0.254747211933136 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01785714365541935 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.12222222238779068 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32184848189353943 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.16316820681095123 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9649801254272461 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003968254197388887 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914575815200806 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1297452300786972 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6174558401107788 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14043210446834564 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9200000166893005 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08737215399742126 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.33309462666511536 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.19197535514831543 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 particle/accuracy_7_of_7‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.011134674772620201 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9853727221488953 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.36723792552948 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009164776653051376 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26205357909202576 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9953324198722839 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2856321632862091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1275210678577423 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1441797912120819 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08847090601921082 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9874438643455505 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.2669812738895416 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.21113678812980652 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9667370319366455 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07589513808488846 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17087262868881226 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1522645503282547 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.22438545525074005 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.1700061708688736 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.16974452137947083 ) Purity/1FRt*SRqqt3FBt/FRt_purity‚0.25 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14925448596477509 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2928960919380188 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21964088082313538 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14975124597549438 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19856323301792145 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16594333946704865 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 particle/accuracy_2_of_7‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.21593062579631805 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020440252497792244 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12126430124044418 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.34387484192848206 particle/accuracy_4_of_5‚1 jet/accuracy_7_of_7‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3380952477455139 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16301319003105164 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9630028009414673 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.059174247086048126 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.22796158492565155 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.172751322388649 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5757575631141663 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚0.5 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.33001792430877686 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 jet/accuracy_6_of_6‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2090158760547638 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14400823414325714 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9773240685462952 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.16974452137947083 ,jet/accuracy_3_of_3‚0.009620742872357368 7validation_average_jet_accuracy‚0.30357128381729126 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -jet/accuracy_4_of_4‚0.0019841270986944437 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_7‚1 trainer/global_step‚1208 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9872013330459595 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.257675439119339 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07892093807458878 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1476937085390091 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_4_of_5‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 particle/accuracy_4_of_6‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06240253150463104 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1524595469236374 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.985213041305542 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.20104163885116577 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1855064034461975 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14264264702796936 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.004576320759952068 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16835732758045197 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1332375556230545 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17428192496299744 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10802468657493591 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_2‚0.0673392191529274 jet/accuracy_6_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2071160078048706 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06680991500616074 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.44736841320991516 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.22577518224716187 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5587201118469238 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.293402761220932 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22750675678253174 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3147244155406952 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12554241716861725 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.20541724562644958 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20807723701000214 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17122112214565277 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14799068868160248 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015325784683228 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.20000000298023224 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 )loss/val_total_loss‚62.96715354045511 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.32959669828414917 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.31992319226264954 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.1818181872367859 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.23275862634181976 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _step‚98 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16491220891475677 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.039509326219558716 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0024719280190765858 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.32692307233810425 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219857305288315 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02338709682226181 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014070585370063782 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.17732080817222595 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16461779177188873 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848107099533081 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3387663960456848 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944029450416565 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_5_of_5‚1 jet/accuracy_5_of_6‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 epoch‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855214953422546 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986714243888855 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23211154341697693 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 *jet/accuracy_1_of_1‚0.3153397738933563 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_5‚0.043189965188503265 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.19035471975803375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2996375262737274 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.20045220851898193 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2631579041481018 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.15207640826702118 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790908694267273 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34878140687942505 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995590448379517 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20218820869922638 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.19135800004005432 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15031619369983673 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990174412727356 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.055854372680187225 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3345472812652588 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.14888009428977966 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.15152506530284882 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11703167110681534 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26796308159828186 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992784857749939 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3015872836112976 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9390590190887451 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20092596113681793 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992784857749939 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02917531318962574 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1372319608926773 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06680991500616074 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26634088158607483 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890487790107727 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2291666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06725849211215973 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2957717776298523 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.16991108655929565 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 (particle/f_score‚0.32148921489715576 particle/accuracy_5_of_7‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.20950856804847717 particle/accuracy_4_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_pur¤ìLÕ~ity‚0.10000000149011612 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005094191525131464 *jet/accuracy_1_of_3‚0.6780730485916138 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2673610746860504 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9867419600486755 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003438554471358657 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3210911750793457 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.2503676414489746 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3282513916492462 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.24285714328289032 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.21913579106330872 *jet/accuracy_1_of_4‚0.7543676495552063 jet/accuracy_4_of_6‚0 ,"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.375 jet/accuracy_2_of_7‚0 _runtime‚ 929.036493865 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953367710113525 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.29436811804771423 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 (particle/accuracy‚0.8322121500968933 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204bÏs]~˜ ‚(:… Æ æÔÜíÍÕ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.545' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.709 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿5™ ‚(: Æ æÔ˜ªÖØ disk.rbd0.out‚0 disk.nvme0n1.in‚ 153.9765625 disk.drbd1777.in‚0% disk./.usageGB‚1108.3511657714844 disk.dm-1.out‚ 34.2421875 disk.sda.out‚ 34.2421875 disk.rbd0.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 153.9765625# disk.nvme0n1.out‚3048.712890625) proc.memory.availableMB‚ 1290.59765625 network.recv‚409148 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 153.9765625 disk.sda2.in‚0 network.sent‚ 1.494258e+06 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.228697776794434 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 34.2421875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 21.123046875% disk.nvme0c0n1.out‚3048.712890625 disk.dm-0.out‚3048.712890625 cpu‚25.401972102861144 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚92.12281703948975# proc.memory.rssMB‚ 2986.58984375) disk./.usagePercent‚74.38237670349461ÕŽ‚“š ‚(:ˆ Õ æÔ½ìÅÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚100.006' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¹Þ› ‚(: Õ æÔæ»Å×) disk./.usagePercent‚74.44437071996364 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 153.9765625% disk.nvme0c0n1.out‚3995.009765625 disk.dm-1.out‚ 34.7578125 disk.sda.out‚ 34.7578125 disk.sda1.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1109.2749214172363 disk.rbd1.in‚0# disk.nvme0n1.out‚3995.009765625 disk.drbd1777.in‚0 disk.dm-0.out‚3995.009765625 network.sent‚ 1.638141e+06 disk.sda2.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd2.out‚ 21.201171875 network.recv‚415879 disk.sda2.out‚0 disk.sda3.out‚ 34.7578125 disk.sda.in‚0 disk.dm-0.in‚ 153.9765625$ memory_percent‚94.14608478546143# proc.memory.rssMB‚ 2864.00390625) proc.memory.percent‚17.48049259185791 disk.rbd3.in‚0 disk.nvme0n1.in‚ 153.9765625( proc.memory.availableMB‚ 959.10546875 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.397367199083675 proc.cpu.threads‚17d½~¹œœ ‚(:‘ ä æÔ–õ¬Ö gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.4' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.880000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¨#  ‚(:• ä æÔòó×Ø disk.drbd1777.out‚0% disk./.usageGB‚1109.2749252319336' disk.nvme0c0n1.out‚4092.91845703125 disk.dm-1.out‚ 35.39453125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2864.00390625 disk.rbd2.in‚0 disk.drbd1777.in‚0" disk.dm-0.out‚4092.91845703125 disk.rbd1.out‚ 0.392578125 disk.rbd3.out‚0 disk.sda3.out‚ 35.39453125) proc.memory.percent‚17.48049259185791 cpu‚25.391765644868272 disk.dm-0.in‚ 153.9765625) disk./.usagePercent‚74.44437097597117! disk.nvme0c0n1.in‚ 153.9765625 disk.nvme0n1.in‚ 153.9765625 disk.sda2.in‚0 disk.sda.out‚ 35.39453125 proc.cpu.threads‚17% disk.nvme0n1.out‚4092.91845703125 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 1.644542e+06 disk.rbd2.out‚ 22.775390625 disk.rbd3.in‚0$ memory_percent‚94.14858818054199 network.recv‚420498 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 958.6953125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0ßM…ã0ž + _wandbruntime‚963  _runtime‚963ñÐÌÄŸ ¾ _runtime‚ 963.27352888 _step‚99 & lr-AdamW/pg1‚0.0001842782273395288 & lr-AdamW/pg2‚0.0001842782273395288 trainer/global_step‚1249 % _timestamp‚1.7884488717191105e+09¢IÔ(È   _runtime‚ 963.27352888 _step‚99 & lr-AdamW/pg1‚0.0001842782273395288 & lr-AdamW/pg2‚0.0001842782273395288 trainer/global_step‚1249 % _timestamp‚1.7884488717191105e+09c0×߸0¡ + _wandbruntime‚963  _runtime‚963©œÅ'¢ ¿O YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚1249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18208374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16701046 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80571816439808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71912909329803 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1047414203392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513Eˆ#É'£ ÃO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚1249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18208374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16701046 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80571816439808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79470248602112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71912909329803 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1047414203392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430d€÷3¯0¤ + _wandbruntime‚963  _runtime‚963ƒ(Ú^¥ ÈM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80634146887680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1102271488000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 963.9711689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23646200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884488724159853e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70924257566720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1048890006016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844Ÿ˜ì$Ò&¦ ÌM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72834855899136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6080083 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80634146887680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1102271488000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 963.9711689 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23646200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1048890006016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884488724159853e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72834855899136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70924257566720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1048890006016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71968629895327 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1048233631232 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79531875399680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1048233631232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944e/ê¸0§ + _wandbruntime‚963  _runtime‚963ªZÊuS¨ Í %loss/total_loss‚60.07780120408296 /loss/FRt1/assignment_loss‚4.177070977432387 0loss/FBt2/assignment_loss‚0.2083835031001584 .loss/FRt3/detection_loss‚2.423488823696971 1loss/SRqqt2/detection_loss‚1.4883431568251628 /loss/FBt1/detection_loss‚5.3373783230781555 .loss/FBt2/detection_loss‚2.052904631517937 1loss/SRqqt2/assignment_loss‚2.166871989243909 .loss/FRt2/detection_loss‚2.361801301349293 0loss/FBt3/assignment_loss‚0.4960907062463775 1loss/SRqqt1/detection_loss‚2.0274750298383286 epoch‚1 trainer/global_step‚1249 0loss/FBt4/assignment_loss‚0.5390913210258312 /loss/FRt1/detection_loss‚2.1573494683418954 /loss/FRt4/detection_loss‚2.7080062569641488 .loss/FBt3/detection_loss‚4.006183169782162 _step‚102 % _timestamp‚1.7884488724183595e+09 /loss/FRt2/assignment_loss‚3.067895974896171 1loss/SRqqt3/assignment_loss‚2.412228382769085 1loss/SRqqt4/assignment_loss‚2.927363178004389 0loss/FBt1/assignment_loss‚0.5992947667837143 1loss/SRqqt3/detection_loss‚2.4993776842242195 .loss/FBt4/detection_loss‚1.706574772511556 /loss/FRt3/assignment_loss‚4.353310123085976 0loss/FRt4/assignment_loss‚4.9497626758203275 1loss/SRqqt4/detection_loss‚3.1977907056393833 _runtime‚ 963.972659118 2loss/SRqqt1/assignment_loss‚2.2137642819054273'À;W© Ñ /loss/FRt3/assignment_loss‚4.353310123085976 0loss/FRt4/assignment_loss‚4.9497626758203275 1loss/SRqqt4/detection_loss‚3.1977907056393833 _runtime‚ 963.972659118 2loss/SRqqt1/assignment_loss‚2.2137642819054273 %loss/total_loss‚60.07780120408296 /loss/FRt1/assignment_loss‚4.177070977432387 0loss/FBt2/assignment_loss‚0.2083835031001584 .loss/FRt3/detection_loss‚2.423488823696971 1loss/SRqqt2/detection_loss‚1.4883431568251628 /loss/FBt1/detection_loss‚5.3373783230781555 .loss/FBt2/detection_loss‚2.052904631517937 1loss/SRqqt2/assignment_loss‚2.166871989243909 .loss/FRt2/detection_loss‚2.361801301349293 0loss/FBt3/assignment_loss‚0.4960907062463775 1loss/SRqqt1/detection_loss‚2.0274750298383286 epoch‚1 trainer/global_step‚1249 0loss/FBt4/assignment_loss‚0.5390913210258312 /loss/FRt1/detection_loss‚2.1573494683418954 /loss/FRt4/detection_loss‚2.7080062569641488 .loss/FBt3/detection_loss‚4.006183169782162 _step‚102 % _timestamp‚1.7884488724183595e+09 /loss/FRt2/assignment_loss‚3.067895974896171 1loss/SRqqt3/assignment_loss‚2.412228382769085 1loss/SRqqt4/assignment_loss‚2.927363178004389 0loss/FBt1/assignment_loss‚0.5992947667837143 1loss/SRqqt3/detection_loss‚2.4993776842242195 .loss/FBt4/detection_loss‚1.706574772511556fìÆ1 ‚(:ˆ ó æÔÈÜÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.8444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü ¿« ‚(:‘ ó æÔ¡¦ÔØ disk.sda.in‚0 disk.sda.out‚ 35.7421875 disk.sda1.in‚0' disk.nvme0c0n1.out‚4093.18408203125$ memory_percent‚94.15888786315918) disk./.usagePercent‚74.44431823841911 disk.rbd0.out‚0# proc.memory.rssMB‚ 2864.00390625 network.recv‚424362 disk.nvme0n1.in‚ 153.9765625 disk.sda1.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚4093.18408203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 957.0078125 disk.rbd1.in‚0 disk.rbd2.out‚ 22.916015625 network.sent‚ 1.649876e+06 disk.drbd1777.in‚0" disk.dm-0.out‚4093.18408203125 disk.sda3.in‚0 cpu‚25.38664421733036 disk.dm-1.out‚ 35.7421875 disk.rbd1.out‚ 0.392578125 disk.rbd3.in‚0 disk.sda3.out‚ 35.7421875) proc.memory.percent‚17.48049259185791% disk./.usageGB‚1109.2741394042969 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 153.9765625 disk.drbd1777.out‚0 disk.dm-0.in‚ 153.9765625 ƒÚt‘¬ ‚(:† ‚¡æÔŒí»Õ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-´ûX­ ‚(:† ‚¡æÔ‹ô‡Ù disk.rbd2.out‚ 22.916015625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda3.out‚36.375) proc.memory.percent‚17.48049259185791 proc.cpu.threads‚17" disk.dm-0.out‚4194.23193359375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.out‚36.375' proc.memory.availableMB‚ 956.7734375) disk./.usagePercent‚74.44450896403212 disk.dm-0.in‚ 153.9765625 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚94.16031837463379% disk./.usageGB‚1109.2769813537598' disk.nvme0c0n1.out‚4194.23193359375 disk.nvme0n1.in‚ 153.9765625 network.sent‚ 1.758293e+06 network.recv‚430662 disk.rbd0.in‚0 disk.dm-1.out‚36.375# proc.memory.rssMB‚ 2864.00390625! disk.nvme0c0n1.in‚ 153.9765625% disk.nvme0n1.out‚4194.23193359375 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.455078125 disk.sda3.in‚0 cpu‚25.381897632135285 disk.rbd2.in‚0eì}B2® - _wandbruntime‚1000  _runtime‚1000ßu©hɯ à ' lr-AdamW/pg1‚0.00018425326560656712 ' lr-AdamW/pg2‚0.00018425326560656712 trainer/global_step‚1299 % _timestamp‚1.7884489090052876e+09 _runtime‚1000.559710505 _step‚103ÉþÐ]Í° Ç trainer/global_step‚1299 % _timestamp‚1.7884489090052876e+09 _runtime‚1000.559710505 _step‚103 ' lr-AdamW/pg1‚0.00018425326560656712 ' lr-AdamW/pg2‚0.00018425326560656712gýœOa2± - _wandbruntime‚1000  _runtime‚1000J¸}ÜÇ'² ÁO _step‚104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73614559964160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83697050358272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75596154802688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6302016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884489090060225e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18909176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76684536808960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24553772 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1088400009216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1088400009216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18909176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76685211173376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74699622674319 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1085045670799 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1144454535168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1144288813056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24553772ÀÛ»i ³ ÅO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24553772 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1088400009216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1088400009216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18909176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11033854 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76685211173376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74699622674319 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1085045670799 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1144454535168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 LB¦¼ˆ bDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1144288813056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24553772 _step‚104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73614559964160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83697050358272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75596154802688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6302016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884489090060225e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18909176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74698951197211 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76684536808960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83697240020480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088h±„¬2´ - _wandbruntime‚1001  _runtime‚10015R!´Ñ&µ ËM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24552190 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24572252 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76742383653376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1001.259412461 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83759570876928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1145158594048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17348939 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚105 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75652507841536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6306493 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17348939 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚105 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75652507841536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6306493 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11442254 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884489465339215e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19609976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19609976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85635298428416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86822412097536 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11442254 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884489465339215e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19609976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19609976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85635298428416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86822412097536 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78470159783936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 % _timestamp‚1.7884489472471213e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1349 _step‚109 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85696949166080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79601021401600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76414670622720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6534271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1126697445379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1130861617664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1038.802149419 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86884923999744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77540713630143 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6534271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1126697445379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1130861617664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1038.802149419 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78470159783936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 % _timestamp‚1.7884489472471213e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78470141780992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1349 _step‚109 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45102232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85696949166080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25478242 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79601021401600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6533828 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77541385107459 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17984693 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76414670622720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422m²gF~2É - _wandbruntime‚1038  _runtime‚1038þ UÊ Ï /loss/FRt1/assignment_loss‚4.107615895131055 .loss/FBt4/detection_loss‚5.154846100664984 % _timestamp‚1.7884489472490535e+09 _runtime‚1038.803403042 /loss/FRt4/detection_loss‚2.8109632611274717 /loss/SRqqt3/detection_loss‚2.39986633800942 /loss/FBt2/detection_loss‚1.2281055563238845 /loss/FRt1/detection_loss‚2.8467547613031723 .loss/FBt3/detection_loss‚3.951750365893046 1loss/FBt2/assignment_loss‚0.33135922538100543 epoch‚1 /loss/FRt2/detection_loss‚2.1805834744553096 1loss/SRqqt4/detection_loss‚2.7075634620019366 /loss/FRt2/assignment_loss‚3.059756918031661 .loss/FRt4/assignment_loss‚5.77455415725708 2loss/SRqqt1/assignment_loss‚2.1525691052277884 1loss/SRqqt2/assignment_loss‚2.427486096819242 2loss/SRqqt3/assignment_loss‚1.7114061366604723 0loss/FBt1/assignment_loss‚1.1947594285011292 /loss/FRt3/assignment_loss‚5.254844597975413 1loss/SRqqt4/assignment_loss‚2.937306685107095 0loss/FBt3/assignment_loss‚0.6922280996892368 1loss/FBt4/assignment_loss‚0.42280886888207847 /loss/FRt3/detection_loss‚2.4151076714197797 1loss/SRqqt1/detection_loss‚2.0169816541450993 .loss/FBt1/detection_loss‚6.028423726558685 trainer/global_step‚1349 1loss/SRqqt2/detection_loss‚1.3454695331553619 %loss/total_loss‚65.15311111972142 _step‚110·ˆEYË Ó 1loss/FBt2/assignment_loss‚0.33135922538100543 epoch‚1 /loss/FRt2/detection_loss‚2.1805834744553096 1loss/SRqqt4/detection_loss‚2.7075634620019366 /loss/FRt2/assignment_loss‚3.059756918031661 .loss/FRt4/assignment_loss‚5.77455415725708 2loss/SRqqt1/assignment_loss‚2.1525691052277884 1loss/SRqqt2/assignment_loss‚2.427486096819242 2loss/SRqqt3/assignment_loss‚1.7114061366604723 0loss/FBt1/assignment_loss‚1.1947594285011292 /loss/FRt3/assignment_loss‚5.254844597975413 1loss/SRqqt4/assignment_loss‚2.937306685107095 0loss/FBt3/assignment_loss‚0.6922280996892368 1loss/FBt4/assignment_loss‚0.42280886888207847 /loss/FRt3/detection_loss‚2.4151076714197797 1loss/SRqqt1/detection_loss‚2.0169816541450993 .loss/FBt1/detection_loss‚6.028423726558685 trainer/global_step‚1349 1loss/SRqqt2/detection_loss‚1.3454695331553619 %loss/total_loss‚65.15311111972142 _step‚110 /loss/FRt1/assignment_loss‚4.107615895131055 .loss/FBt4/detection_loss‚5.154846100664984 % _timestamp‚1.7884489472490535e+09 _runtime‚1038.803403042 /loss/FRt4/detection_loss‚2.8109632611274717 /loss/SRqqt3/detection_loss‚2.39986633800942 /loss/FBt2/detection_loss‚1.2281055563238845 /loss/FRt1/detection_loss‚2.8467547613031723 .loss/FBt3/detection_loss‚3.951750365893046nz7®(žÌ ‚(:“ ¾¡æÔïÔÛÕ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚104.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.962799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:@®¥Í ‚(:û ¾¡æÔº¢ø× disk.nvme0c0n1.in‚156.5$ proc.memory.availableMB‚955.9375 cpu‚25.363671254697746 disk.dm-0.in‚156.5 disk.sda1.out‚0 disk.rbd2.out‚24.66650390625$ disk.nvme0c0n1.out‚ 4297.98046875" disk.nvme0n1.out‚ 4297.98046875 disk.dm-1.out‚ 39.26171875) disk./.usagePercent‚74.44453840489857 disk.sda.in‚0 disk.sda.out‚ 39.26171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚156.5 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.out‚ 4297.98046875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.48049259185791% disk./.usageGB‚1109.2774200439453 disk.rbd1.in‚0 disk.rbd1.out‚ 0.55029296875 disk.rbd0.in‚0$ memory_percent‚94.16542053222656# proc.memory.rssMB‚ 2864.00390625 network.sent‚ 1.884153e+06 network.recv‚449172 disk.sda3.in‚0 disk.sda3.out‚ 39.26171875 disk.rbd0.out‚09 ’۞Π‚(:“ Í¡æÔÚ·–Ö gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.480399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ9UÏ ‚(: Í¡æÔÉôÄØ disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 39.92578125 disk.rbd2.in‚0 network.sent‚ 1.992007e+06 disk.drbd1777.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 956.2734375 proc.cpu.threads‚17% disk./.usageGB‚1109.2781028747559 network.recv‚455487 disk.dm-0.in‚ 157.46875! disk.dm-0.out‚4396.1142578125 disk.rbd1.out‚ 0.55029296875 disk.sda3.in‚0 disk.rbd2.out‚ 24.7685546875& disk.nvme0c0n1.out‚4396.1142578125$ disk.nvme0n1.out‚4396.1142578125) disk./.usagePercent‚74.44458423024719 disk.dm-1.out‚ 39.92578125 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 157.46875# proc.memory.rssMB‚ 2864.00390625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 157.46875 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚94.16337013244629) proc.memory.percent‚17.48049259185791 disk.sda.out‚ 39.92578125 disk.rbd0.out‚0 cpu‚25.359328552616148mB3v2Ð - _wandbruntime‚1075  _runtime‚1075¿q¦6ÉÑ Ã _step‚111 ' lr-AdamW/pg1‚0.00018414576391025274 ' lr-AdamW/pg2‚0.00018414576391025274 trainer/global_step‚1399 % _timestamp‚1.7884489837969222e+09 _runtime‚1075.351477211ä]6ÍÒ Ç ' lr-AdamW/pg1‚0.00018414576391025274 ' lr-AdamW/pg2‚0.00018414576391025274 trainer/global_step‚1399 % _timestamp‚1.7884489837969222e+09 _runtime‚1075.351477211 _step‚111o5ª<2Ó - _wandbruntime‚1075  _runtime‚1075ÅÓêµÇ'Ô ÁO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89947974289920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88717835311616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88717859251712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _runtime‚1075.352437721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26405876 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82402486669824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20310774 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20310774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11850652Óó2qÕ ÅO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _runtime‚1075.352437721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26405876 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82402486669824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20310774 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20310774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11850652 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89947974289920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88717835311616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88717859251712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0põ°¡Õ2Ö - _wandbruntime‚1076  _runtime‚1076jÝ5ÔÐ&× ÊM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788448984481572e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80327426974867 _runtime‚1076.036587064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 _step‚113 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20324790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26404294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81287811726336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82459659149824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81287811726336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11858820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230797279744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82459659149824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81287811726336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11858820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 _step‚113 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20324790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26404294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20324790 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81287811726336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392q,n2Ù - _wandbruntime‚1076  _runtime‚1076ú@X÷QÚ Ë /loss/FRt4/assignment_loss‚5.165322226862753 1loss/SRqqt3/assignment_loss‚1.895438212280472 /loss/FBt2/assignment_loss‚0.356619024891848 .loss/FRt4/detection_loss‚2.963675768144669 1loss/SRqqt2/detection_loss‚1.5512705991665523 /loss/FRt1/assignment_loss‚4.565619673047747 1loss/SRqqt1/assignment_loss‚1.937406817487642 0loss/SRqqt3/detection_loss‚1.998991820961237 1loss/SRqqt4/detection_loss‚2.4929779343532794 trainer/global_step‚1399 $ _timestamp‚1.788448984483796e+09 /loss/FRt1/detection_loss‚2.3756260117705987 /loss/FRt2/assignment_loss‚2.725068326625559 /loss/FRt2/detection_loss‚1.8204005712436304 .loss/FBt1/detection_loss‚4.610428512096405 /loss/FBt3/detection_loss‚2.9673770546913145 _runtime‚1076.038106657 _step‚114 /loss/FBt2/detection_loss‚0.6674123462671983 epoch‚1 .loss/FRt3/detection_loss‚2.338288718745822 1loss/SRqqt2/assignment_loss‚2.023805929819743 0loss/FBt3/assignment_loss‚0.7686774142086505 0loss/FBt4/assignment_loss‚0.2896316318451362 0loss/SRqqt1/detection_loss‚2.075329089865965 %loss/total_loss‚57.66891702246424 0loss/FRt3/assignment_loss‚3.9157946478752863 2loss/SRqqt4/assignment_loss‚3.0922836137540415 0loss/FBt1/assignment_loss‚0.8444791585206985 .loss/FBt4/detection_loss‚4.226991917937994$|w†UÛ Ï /loss/FRt2/assignment_loss‚2.725068326625559 /loss/FRt2/detection_loss‚1.8204005712436304 .loss/FBt1/detection_loss‚4.610428512096405 /loss/FBt3/detection_loss‚2.9673770546913145 _runtime‚1076.038106657 _step‚114 /loss/FBt2/detection_loss‚0.6674123462671983 epoch‚1 .loss/FRt3/detection_loss‚2.338288718745822 1loss/SRqqt2/assignment_loss‚2.023805929819743 0loss/FBt3/assignment_loss‚0.7686774142086505 0loss/FBt4/assignment_loss‚0.2896316318451362 0loss/SRqqt1/detection_loss‚2.075329089865965 %loss/total_loss‚57.66891702246424 0loss/FRt3/assignment_loss‚3.9157946478752863 2loss/SRqqt4/assignment_loss‚3.0922836137540415 0loss/FBt1/assignment_loss‚0.8444791585206985 .loss/FBt4/detection_loss‚4.226991917937994 /loss/FRt4/assignment_loss‚5.165322226862753 1loss/SRqqt3/assignment_loss‚1.895438212280472 /loss/FBt2/assignment_loss‚0.356619024891848 .loss/FRt4/detection_loss‚2.963675768144669 1loss/SRqqt2/detection_loss‚1.5512705991665523 /loss/FRt1/assignment_loss‚4.565619673047747 1loss/SRqqt1/assignment_loss‚1.937406817487642 0loss/SRqqt3/detection_loss‚1.998991820961237 1loss/SRqqt4/detection_loss‚2.4929779343532794 trainer/global_step‚1399 $ _timestamp‚1.788448984483796e+09 /loss/FRt1/detection_loss‚2.3756260117705987r5ç*žÜ ‚(:“ Ü¡æÔº½ÊÕ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.973' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.794600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰[GòûÝ ‚(:ð Ü¡æÔ¼Ù—Ø disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 157.46875 disk.rbd2.in‚0 memory_percent‚ 94.1650390625 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 proc.memory.availableMB‚956 network.recv‚460171) disk./.usagePercent‚74.44456579770473 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 4398.1328125 disk.dm-0.in‚ 157.46875 disk.dm-0.out‚ 4398.1328125 disk.dm-1.out‚ 40.7578125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2864.00390625) proc.memory.percent‚17.48049259185791 cpu‚25.35514908813 disk.sda3.out‚ 40.7578125 network.sent‚ 1.998299e+06 disk.sda.out‚ 40.7578125 disk.rbd0.out‚0 disk.rbd2.out‚ 24.7685546875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 157.46875! disk.nvme0n1.out‚ 4398.1328125% disk./.usageGB‚1109.2778282165527¯„¤àžÞ ‚(:“ ë¡æÔµÂÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.587' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.517399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NDY“ß ‚(:Š ë¡æÔ¸–‡Ø disk.drbd1777.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 158.265625 disk.nvme0n1.in‚ 158.265625 disk.dm-0.in‚ 158.265625" disk.dm-0.out‚4495.92236328125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.out‚41.34375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.out‚41.34375 disk.sda1.out‚0 disk.rbd0.out‚0 cpu‚25.351070511506304% disk./.usageGB‚1109.2779426574707 disk.dm-1.in‚0# proc.memory.rssMB‚ 2864.00390625 disk.dm-1.out‚41.34375$ memory_percent‚94.16558742523193( proc.memory.availableMB‚ 955.91015625 disk.rbd2.in‚0' disk.nvme0c0n1.out‚4495.92236328125) disk./.usagePercent‚74.44457347793076 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0% disk.nvme0n1.out‚4495.92236328125 network.sent‚ 2.105765e+06) proc.memory.percent‚17.48049259185791 network.recv‚466045 disk.rbd1.out‚ 0.61328125 disk.rbd2.out‚ 24.9091796875ÈŠŒžà ‚(:“ ú¡æÔ—³ÐÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.482' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.296400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù&§á ‚(:ƒ ú¡æÔ¤áý× disk.sda3.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0% disk./.usageGB‚1109.2779426574707 disk.rbd1.in‚0 disk.sda3.out‚ 41.84765625 disk.rbd0.in‚0 disk.rbd2.out‚ 24.9091796875 disk.sda2.in‚0 network.sent‚ 2.112012e+06 disk.nvme0c0n1.in‚ 158.265625" disk.nvme0c0n1.out‚ 4496.671875 disk.dm-1.out‚ 41.84765625( proc.memory.availableMB‚ 955.92578125) proc.memory.percent‚17.48049259185791 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 158.265625 disk.dm-0.out‚ 4496.671875) disk./.usagePercent‚74.44457347793076 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 4496.671875 disk.drbd1777.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 158.265625 network.recv‚470454 disk.sda.out‚ 41.84765625 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2864.00390625 disk.rbd2.in‚0 disk.drbd1777.out‚0# memory_percent‚94.1654920578003 cpu‚25.347096924088554Kçî2â - _wandbruntime‚1112  _runtime‚1112ÁRýÉã à ' lr-AdamW/pg2‚0.00018406324635009241 trainer/global_step‚1449 % _timestamp‚1.7884490207431087e+09 _runtime‚1112.297602803 _step‚115 ' lr-AdamW/pg1‚0.00018406324635009241'½ççÍä Ç _runtime‚1112.297602803 _step‚115 ' lr-AdamW/pg1‚0.00018406324635009241 ' lr-AdamW/pg2‚0.00018406324635009241 trainer/global_step‚1449 % _timestamp‚1.7884490207431087e+09s°&„2å - _wandbruntime‚1112  _runtime‚1112ˆ&òðˆæ ¿O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884490207439919e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_star&þ¹%=$t/requested_bytes.all.peak‚ 9643699017 _runtime‚ 1112.2986729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1211357426688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21011574 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12259054 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6985431 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91800396134912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1449 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85260450053632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27311892 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21011574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27331928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93073290703872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27311892 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12259052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21011574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206897251275 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27331928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93073290703872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1272918509056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 _runtime‚ 1112.2986729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1212013788160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83057748276543 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1211357426688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416t2ÒÈl2è - _wandbruntime‚1112  _runtime‚1112*ŒDÛÑ&é ËM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1212176854528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6989537 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1273939154944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93135794281472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884490214345598e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85318296898048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 trainer/global_step‚1449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1212833229312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48355936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 99555850245ŸƒaÔ ê ÏM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83113468842275 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85318296898048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21025590 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85317622520320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YFDeviceStatsMonitor.on_train_batch_en(«êd/requested_bytes.large_pool.freed‚81905083678720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 trainer/global_step‚1449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1212833229312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48355936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85318296898048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1212176854528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21025590 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6989537 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1273939154944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1273771348992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81905100718080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12267222 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93135794281472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884490214345598e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27330346 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 trainer/global_step‚1499 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28257980 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84595386076160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1247732590683 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49950318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚1149.622635117 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94882933018112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252343232512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88119762166272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344$K%¤%÷ ÅO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252343232512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88119762166272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252343232512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88119087801856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21712374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 % _timestamp‚1.7884490580679603e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84595403115520 trainer/global_step‚1499 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28257980 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84595386076160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1247732590683 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49950318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚1149.622635117 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85843118666843 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94882933018112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated\4'‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0xPØ"¦2ø - _wandbruntime‚1150  _runtime‚1150<Û¬@Ñ&ù ËM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12675622 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚121 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253162660352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1316977635840 _runtime‚1150.358678461 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1253819035136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28276460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ÚÞ’JÕ&ú ÏM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12675622 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚121 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12675620 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253162660352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1316977635840 _runtime‚1150.358678461 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1253819035136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28276460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430y$¯»ì2û - _wandbruntime‚1150  _runtime‚1150¥² —Sü Í .loss/FBt1/detection_loss‚4.338991602261861 /loss/FBt3/detection_loss‚3.6746859163045884 /loss/FRt2/assignment_loss‚2.128694488840588 1loss/FBt1/assignment_loss‚0.12517737849460295 0loss/FRt3/assignment_loss‚4.5883999183064414 0loss/FBt3/assignment_loss‚0.7049923606216908 .loss/FRt4/detection_loss‚3.068190813064575 1loss/SRqqt2/detection_loss‚1.3885428672656417 .loss/FBt4/detection_loss‚3.972143765538931 /loss/FRt4/assignment_loss‚5.120907187461853 1loss/FBt2/assignment_loss‚0.27212540810193164 /loss/FRt2/detection_loss‚1.8686582370329712 1loss/SRqqt1/detection_loss‚2.3327248983439945 /loss/SRqqt4/detection_loss‚2.90192086800285 /loss/FBt2/detection_loss‚1.0969439154905085 %loss/total_loss‚58.58421944761091 1loss/FBt4/assignment_loss‚0.38598817400759344 1loss/SRqqt3/assignment_loss‚2.353253520549612 epoch‚1 _runtime‚1150.360316909 _step‚122 $ _timestamp‚1.788449058805957e+09 /loss/FRt1/assignment_loss‚4.819263861459844 1loss/SRqqt1/assignment_loss‚2.007599014611471 1loss/SRqqt4/assignment_loss‚2.994889285253442 .loss/FRt1/detection_loss‚2.662020683288574 trainer/global_step‚1499 2loss/SRqqt2/assignment_loss‚1.4587109148064081 .loss/FRt3/detection_loss‚2.262684006066549 0loss/SRqqt3/detection_loss‚2.056710362434387 /’Wý Ñ 1loss/SRqqt3/assignment_loss‚2.353253520549612 epoch‚1 _runtime‚1150.360316909 _step‚122 $ _timestamp‚1.788449058805957e+09 /loss/FRt1/assignment_loss‚4.819263861459844 1loss/SRqqt1/assignment_loss‚2.007599014611471 1loss/SRqqt4/assignment_loss‚2.994889285253442 .loss/FRt1/detection_loss‚2.662020683288574 trainer/global_step‚1499 2loss/SRqqt2/assignment_loss‚1.4587109148064081 .loss/FRt3/detection_loss‚2.262684006066549 0loss/SRqqt3/detection_loss‚2.056710362434387 .loss/FBt1/detection_loss‚4.338991602261861 /loss/FBt3/detection_loss‚3.6746859163045884 /loss/FRt2/assignment_loss‚2.128694488840588 1loss/FBt1/assignment_loss‚0.12517737849460295 0loss/FRt3/assignment_loss‚4.5883999183064414 0loss/FBt3/assignment_loss‚0.7049923606216908 .loss/FRt4/detection_loss‚3.068190813064575 1loss/SRqqt2/detection_loss‚1.3885428672656417 .loss/FBt4/detection_loss‚3.972143765538931 /loss/FRt4/assignment_loss‚5.120907187461853 1loss/FBt2/assignment_loss‚0.27212540810193164 /loss/FRt2/detection_loss‚1.8686582370329712 1loss/SRqqt1/detection_loss‚2.3327248983439945 /loss/SRqqt4/detection_loss‚2.90192086800285 /loss/FBt2/detection_loss‚1.0969439154905085 %loss/total_loss‚58.58421944761091 1loss/FBt4/assignment_loss‚0.38598817400759344z¥ I–“þ ‚(:ˆ §¢æÔõ¾ÃÕ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÅ*cÿ ‚(:Š §¢æÔ½ç¼Ø disk.drbd1777.in‚0# proc.memory.rssMB‚ 2864.00390625 disk.dm-0.out‚4595.251953125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.in‚0# disk.nvme0n1.out‚4595.251953125) proc.memory.percent‚17.48049259185791 disk.sda3.out‚ 43.671875 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.73046875 disk.rbd1.out‚ 0.61328125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 158.38671875 disk.drbd1777.out‚0# memory_percent‚99.9589204788208 disk.sda.in‚0$ disk./.usageGB‚1109.277286529541 disk.rbd0.out‚0 network.sent‚ 2.231566e+06 network.recv‚484518 disk.dm-0.in‚ 158.38671875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0% disk.nvme0c0n1.out‚4595.251953125) disk./.usagePercent‚74.44452944463487 disk.dm-1.out‚ 43.671875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 158.38671875 cpu‚25.335962966744855 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.out‚ 43.671875 disk.rbd2.out‚26.71044921875O«’g€‚(:’ ¶¢æÔ©ëÿÕ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.581999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09[}‚(: ¶¢æÔ¼Õ°Ø network.sent‚ 2.340152e+06 disk.rbd3.out‚0 disk.sda3.out‚ 44.0546875 disk.sda2.out‚0& proc.memory.availableMB‚ 6.58984375 network.recv‚490880) disk./.usagePercent‚74.44453328474788 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 158.38671875% disk.nvme0n1.out‚4693.56787109375 disk.dm-1.out‚ 44.0546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 158.38671875 disk.sda2.in‚0) proc.memory.percent‚17.48049259185791' disk.nvme0c0n1.out‚4693.56787109375" disk.dm-0.out‚4693.56787109375 disk.dm-1.in‚0 cpu‚25.332379423940242 disk./.usageGB‚ 1109.27734375 disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0 disk.sda.out‚ 44.0546875 disk.rbd2.out‚26.72607421875 disk.drbd1777.in‚0 disk.dm-0.in‚ 158.38671875$ memory_percent‚99.95977878570557 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2864.00390625 æ!’‚‚(:‡ Å¢æÔû”¸Õ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚96.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²cƒ‚(: Å¢æÔ•¨Ø network.recv‚494803 disk.sda.out‚ 44.5234375 disk.rbd0.in‚0( disk./.usagePercent‚74.4445371248609$ disk./.usageGB‚1109.277400970459# disk.nvme0n1.out‚4693.986328125 disk.sda1.out‚0 disk.dm-0.in‚ 158.38671875$ memory_percent‚99.95672702789307 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0 disk.rbd2.out‚26.78857421875% disk.nvme0c0n1.out‚4693.986328125 cpu‚25.329090465745583 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 158.38671875 disk.dm-1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2864.00390625 disk.sda3.in‚0 disk.sda3.out‚ 44.5234375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.dm-0.out‚4693.986328125 disk.rbd1.in‚0& proc.memory.availableMB‚ 7.08984375) proc.memory.percent‚17.48049259185791" disk.nvme0c0n1.in‚ 158.38671875 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 44.5234375 network.sent‚ 2.345853e+06ªç2„- _wandbruntime‚1186  _runtime‚1186*|ðÈ…Â trainer/global_step‚1549 % _timestamp‚1.7884490953300726e+09 _runtime‚ 1186.88452649 _step‚123 ' lr-AdamW/pg1‚0.00018384076378486115 ' lr-AdamW/pg2‚0.00018384076378486115_ýÆXÌ†Æ ' lr-AdamW/pg1‚0.00018384076378486115 ' lr-AdamW/pg2‚0.00018384076378486115 trainer/global_step‚1549 % _timestamp‚1.7884490953300726e+09 _runtime‚ 1186.88452649 _step‚123{ÞŸV2‡- _wandbruntime‚1186  _runtime‚1186qD ›¥ˆÁO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22413176 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90977725550080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚422 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89684396511744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1293329038336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20518551 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 trainer/global_step‚1549 _step‚124 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20518973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13075854 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1288567930091 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87340609643520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29163996 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29184032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51577170 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87340592604160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \IDeviceStatsMonitor.on_tó8Á"rain_batch_start/active_bytes.large_pool.allocated‚89684414514688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29163996 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22413174 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1358922105344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884490953311906e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99324547243008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97965469901312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90978399914496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884490953311906e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22413174 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99324547243008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97965469901312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90978399914496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1293985399808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7443119 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89684396511744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1293329038336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20518551 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 trainer/global_step‚1549 _step‚124 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20518973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7442699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096|Œo³:2Š- _wandbruntime‚1187  _runtime‚1187û„_ Ï&‹ÉM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788449096044541e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7447206 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22427190 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 _runtime‚ 1187.59959034 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29202512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640áê,±ïŒÍM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29182450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 _runtime‚ 1187.59959034 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29202512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89740767553536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1359762685952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13084020 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788449096044541e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7447206 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22427190 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29182450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_èa›3ä end/active.all.allocated‚51629704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29202512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101047982844416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102450009562624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101048006784512 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13484254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1402002778112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90085799132160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30110084 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92502066457088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401841250304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23113974 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30090048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102449824094720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53204022 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30090048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884491328221033e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23113976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1334971205632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1329403269499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92502048454144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Èî±¼Ð'™ÊO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23113974 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30090048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102449824094720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53204022 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30090048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884491328221033e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23113976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30110084 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1334971205632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1329403269499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92502048454144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21155701 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1334314844160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101047982844416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102450009562624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101048006784512 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92502066457088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13484254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1402002778112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90085799132160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30110084 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93837037662720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92502066457088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401841250304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92502048454144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108€ÉW+†2š- _wandbruntime‚1225  _runtime‚1225þ¡m¥¾ ›ÎM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7675950 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1335790646784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92558401492992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30108502 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93894210142720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53236492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1335790646784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53256556 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53236492 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370L.YÙ&œÓM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102512530442752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23127990 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 trainer/global_step‚1599 % _timestamp‚1.7884491335164964e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93894210142720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53236492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1335790646784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53256556 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53236492 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7675950 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1335790646784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92558401492992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30108502 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492420 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54830874 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375957011456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92831005660160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23814776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1261.694445652 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23814776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96695001046528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21790669 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7898480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95319718399488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 ^JDeviceStatsMonitor.on_train_batch_st‚fnÎart/inactive_split_bytes.all.allocated‚105575662560768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445118893056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370238608907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚132 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96695675410944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 t?Ï'«ÉO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23814774 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104130519727616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23814774 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23814776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96695001046528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21790669 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7898480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95319718399488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1444942685184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105575662560768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445118893056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370238608907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚132 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96695675410944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31036136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104130543667712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54830874 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375957011456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31036136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92831005660160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23814776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1261.694445652 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95376053435392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96752847890944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31034554 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371709481827 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚133 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256964834591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105637962808832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92885909790720 trainer/global_step‚1649 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23828790 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96752173513216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23828792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376776452608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚1262.396591239 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105638158749184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414ó¥­Ù&®ÓM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95376071438336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31054616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31034554 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104192194405376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371709481827 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚133 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1445792343552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94256964834591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105637962808832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92885909790720 trainer/global_step‚1649 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23828790 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96752173513216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23828792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376776452608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚1262.396591239 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105638158749184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95376053435392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92885926830080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13900822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95376053435392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1371055043871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96752847890944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0…^„"2¯- _wandbruntime‚1262  _runtime‚1262‡ÿécé°Ï .loss/FRt3/detection_loss‚2.288068417991911 1loss/SRqqt2/detection_loss‚1.3375507452262518 /loss/FBt4/detection_loss‚3.2214127884191623 1loss/FBt1/assignment_loss‚0.42613635063171384 /loss/FBt3/assignment_loss‚0.584573226980865 1loss/SRqqt4/detection_loss‚3.2397952347993852 $ _timestamp‚1.788449170843336e+09 _step‚134 0loss/FRt3/assignment_loss‚3.9473188860075816 /loss/FRt4/assignment_loss‚4.616708465244459 1loss/SRqqt3/detection_loss‚¢z6l1.8891358253907184 .loss/FBt2/detection_loss‚1.122427530810595 1loss/FBt2/assignment_loss‚0.21778831966653037 .loss/FBt3/detection_loss‚4.125404313206673 _runtime‚1262.397624166 2loss/SRqqt3/assignment_loss‚1.9610310123891246 1loss/SRqqt1/detection_loss‚1.8766474952852283 #loss/total_loss‚55.179496384716 0loss/FRt2/assignment_loss‚2.8361993933978833 trainer/global_step‚1649 /loss/FRt2/detection_loss‚1.8242170740115016 2loss/SRqqt1/assignment_loss‚1.5691847416261833 2loss/SRqqt2/assignment_loss‚1.8445707297860048 2loss/SRqqt4/assignment_loss‚2.3176348879933357 .loss/FBt4/assignment_loss‚0.65138795180277 .loss/FBt1/detection_loss‚3.659437155723572 epoch‚1 .loss/FRt4/detection_loss‚3.168816353963769 /loss/FRt1/detection_loss‚2.6679645741269704 0loss/FRt1/assignment_loss‚3.7860849102338157Ÿ³dZ±Ô 1loss/FBt1/assignment_loss‚0.42613635063171384 /loss/FBt3/assignment_loss‚0.584573226980865 1loss/SRqqt4/detection_loss‚3.2397952347993852 $ _timestamp‚1.788449170843336e+09 _step‚134 0loss/FRt3/assignment_loss‚3.9473188860075816 /loss/FRt4/assignment_loss‚4.616708465244459 1loss/SRqqt3/detection_loss‚1.8891358253907184 .loss/FBt2/detection_loss‚1.122427530810595 1loss/FBt2/assignment_loss‚0.21778831966653037 .loss/FBt3/detection_loss‚4.125404313206673 _runtime‚1262.397624166 2loss/SRqqt3/assignment_loss‚1.9610310123891246 1loss/SRqqt1/detection_loss‚1.8766474952852283 #loss/total_loss‚55.179496384716 0loss/FRt2/assignment_loss‚2.8361993933978833 trainer/global_step‚1649 /loss/FRt2/detection_loss‚1.8242170740115016 2loss/SRqqt1/assignment_loss‚1.5691847416261833 2loss/SRqqt2/assignment_loss‚1.8445707297860048 2loss/SRqqt4/assignment_loss‚2.3176348879933357 .loss/FBt4/assignment_loss‚0.65138795180277 .loss/FBt1/detection_loss‚3.659437155723572 epoch‚1 .loss/FRt4/detection_loss‚3.168816353963769 /loss/FRt1/detection_loss‚2.6679645741269704 0loss/FRt1/assignment_loss‚3.7860849102338157 .loss/FRt3/detection_loss‚2.288068417991911 1loss/SRqqt2/detection_loss‚1.3375507452262518 /loss/FBt4/detection_loss‚3.2214127884191623†¨FzÈž²‚(:“ Ÿ£æÔì‰ÀÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.327' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.265400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rõg³‚(:‰ Ÿ£æÔ¬ìå× disk.nvme0n1.in‚ 158.3984375# proc.memory.rssMB‚ 2864.00390625 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda.out‚ 47.80859375% disk.nvme0n1.out‚5152.13720703125 cpu‚25.30999138403961 network.sent‚ 2.5843e+06 disk.dm-1.in‚0 disk.sda3.out‚ 47.80859375 network.recv‚523354 disk.rbd3.in‚0$ memory_percent‚99.96261596679688! proc.memory.availableMB‚6.125 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 28.6796875 disk.dm-1.out‚ 47.80859375 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚5152.13720703125% disk./.usageGB‚1108.2417945861816 disk.dm-0.in‚ 158.3984375 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.48049259185791! disk.nvme0c0n1.in‚ 158.3984375) disk./.usagePercent‚74.37503671148042" disk.dm-0.out‚5152.13720703125 disk.sda2.out‚0 disk.rbd0.out‚0þ´ó“´‚(:ˆ ®£æÔŽáÏÕ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~dðVµ‚(:Ž ®£æÔÔœñ× network.recv‚529772% disk./.usageGB‚1108.2418594360352 disk.rbd0.in‚0 disk.rbd2.out‚ 30.26953125 disk.dm-1.out‚ 48.4453125 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 158.3984375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚5152.53564453125% disk.nvme0n1.out‚5152.53564453125" disk.dm-0.out‚5152.53564453125) proc.memory.percent‚17.48049259185791 proc.cpu.threads‚17 disk.sda2.out‚0 disk.nvme0n1.in‚ 158.3984375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.30702353059247) disk./.usagePercent‚74.37504106360849 disk.rbd1.in‚0 disk.sda.out‚ 48.4453125 disk.sda1.out‚0$ memory_percent‚99.96263980865479& proc.memory.availableMB‚ 6.12109375 disk.dm-1.in‚0 network.sent‚ 2.692673e+06 disk.sda2.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda3.out‚ 48.4453125 disk.dm-0.in‚ 158.3984375# proc.memory.rssMB‚ 2864.00390625}©ú2¶- _wandbruntime‚1299  _runtime‚1299+¶éÉ·Ã % _timestamp‚1.7884492078091223e+09 _runtime‚1299.363514446 _step‚135 ' lr-AdamW/pg1‚0.00018336379893800085 ' lr-AdamW/pg2‚0.00018336379893800085 trainer/global_step‚1699¹Ù¹ Î¸È % _timestamp‚1.7884492078091223e+09 _runtime‚1299.363514446 _step‚135 ' lr-AdamW/pg1‚0.00018336379893800085 ' lr-AdamW/pg2‚0.00018336379893800085 trainer/global_step‚1699‡˜½$A2¹- _wandbruntime‚1299  _runtime‚1299á ¢°Ë'ºÅO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98137352338944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96987957613583 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24515576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 _step‚136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884492078098269e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96987286136475 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14301052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99554313159168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320ó5ÇMÐ'»ÊO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98137370341888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98137352338944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14301052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99554313159168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1416942817280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98137352338944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108701195759104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96987957613583 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107213080550912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24515576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 _step‚136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884492078098269e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96987286136475 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1488115208192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1411728386063 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ˆNïb2¼- _wandbruntime‚1300  _runtime‚1300xí¯²½ÏM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1488962415104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 F#ÆÐÚ&¾ÔM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107274731288576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31980668 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590‰v9v(2¿- _wandbruntime‚1300  _runtime‚1300ýÝYÀÓ 2loss/SRqqt2/assignment_loss‚1.5847213480621576 /loss/FRt1/detection_loss‚2.3062629668662944 .loss/FRt4/detection_loss‚3.040213616689046 1loss/SRqqt2/detection_loss‚1.5136167074952807 /loss/FBt3/detection_loss‚3.2599167029062905 trainer/global_step‚1699 0loss/FRt2/assignment_loss‚2.6958268577769653 /loss/FRt3/detection_loss‚2.3765538533528647 1loss/SRqqt4/detection_loss‚3.3444729412303253 epoch‚1 % _timestamp‚1.7884492085191827e+09 _step‚138 .loss/FRt1/assignment_loss‚3.82190270225207 /loss/FRt4/assignment_loss‚5.371580155690511 0loss/FBt4/assignment_loss‚0.4038755215291227 0loss/SRqqt3/detection_loss‚2.209039056243805 _runtime‚1300.073477389 0loss/FRt3/assignment_loss‚4.3241855303446455 2loss/SRqqt4/assignment_loss‚2.5918337737812713 /loss/FBt2/detection_loss‚0.6785630784879095 1loss/SRqqt1/assignment_loss‚1.966744136697841 1loss/FBt1/assignment_loss‚0.23585495119914412 /loss/FRt2/detection_loss‚1.8112746600882481 .loss/FBt1/detection_loss‚3.142972926298777 .loss/FBt4/detection_loss‚3.273265337944031 %loss/total_loss‚54.63225440878608 2loss/SRqqt3/assignment_loss‚1.8433806545172746 1loss/FBt2/assignment_loss‚0.23447020380260106 0loss/FBt3/assignment_loss‚0.6867850474599335 1loss/SRqqt1/detection_loss‚1.9149416780696724F'R:^ÁØ % _timestamp‚1.7884492085191827e+09 _step‚138 .loss/FRt1/assignment_loss‚3.82190270225207 /loss/FRt4/assignment_loss‚5.371580155690511 0loss/FBt4/assignment_loss‚0.4038755215291227 0loss/SRqqt3/detection_loss‚2.209039056243805 _runtime‚1300.073477389 0loss/FRt3/assignment_loss‚4.3241855303446455 2loss/SRqqt4/assignment_loss‚2.5918337737812713 /loss/FBt2/detection_loss‚0.6785630784879095 1loss/SRqqt1/assignment_loss‚1.966744136697841 1loss/FBt1/assignment_loss‚0.23585495119914412 /loss/FRt2/detection_loss‚1.8112746600882481 .loss/FBt1/detection_loss‚3.142972926298777 .loss/FBt4/detection_loss‚3.273265337944031 %loss/total_loss‚54.63225440878608 2loss/SRqqt3/assignment_loss‚1.8433806545172746 1loss/FBt2/assignment_loss‚0.23447020380260106 0loss/FBt3/assignment_loss‚0.6867850474599335 1loss/SRqqt1/detection_loss‚1.9149416780696724 2loss/SRqqt2/assignment_loss‚1.5847213480621576 /loss/FRt1/detection_loss‚2.3062629668662944 .loss/FRt4/detection_loss‚3.040213616689046 1loss/SRqqt2/detection_loss‚1.5136167074952807 /loss/FBt3/detection_loss‚3.2599167029062905 trainer/global_step‚1699 0loss/FRt2/assignment_loss‚2.6958268577769653 /loss/FRt3/detection_loss‚2.3765538533528647 1loss/SRqqt4/detection_loss‚3.3444729412303253 epoch‚1Š+܆“‚(:ˆ ½£æÔÁöÝÕ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.258' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y'çÂ(:‰ ½£æÔÛËö× network.sent‚ 2.698303e+06 disk.rbd0.out‚0# proc.memory.rssMB‚ 2864.00390625 disk.sda.out‚49.09375 disk.sda3.out‚49.09375! disk.nvme0c0n1.in‚ 158.3984375' disk.nvme0c0n1.out‚5152.83642578125 disk.nvme0n1.in‚ 158.3984375 proc.cpu.threads‚17) disk./.usagePercent‚74.37499421422973 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 30.33203125" disk.dm-0.out‚5152.83642578125 cpu‚25.304124424813722 disk.sda3.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚5152.83642578125 network.recv‚534020$ memory_percent‚99.96106624603271) proc.memory.percent‚17.48049259185791 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 158.3984375& proc.memory.availableMB‚ 6.37890625% disk./.usageGB‚1108.2411613464355 disk.dm-1.out‚49.09375 disk.rbd1.in‚0 disk.drbd1777.out‚0]Ø$s“Ä‚(:ˆ Ì£æÔͽîÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.763' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P8n}Å‚(:‹ Ì£æÔè™ßØ disk.dm-0.in‚ 158.3984375 disk.sda.out‚ 49.5859375# proc.memory.rssMB‚ 2864.00390625 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 49.5859375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0% proc.memory.availableMB‚ 6.3203125 cpu‚25.301259723754608! disk.nvme0c0n1.in‚ 158.3984375 disk.rbd2.in‚0 disk.rbd2.out‚ 30.34765625& disk.nvme0c0n1.out‚5153.4111328125 disk.sda3.in‚0$ disk.nvme0n1.out‚5153.4111328125 disk.sda2.out‚0 network.sent‚ 2.806793e+06 network.recv‚540520 disk.sda1.out‚0 disk.nvme0n1.in‚ 158.3984375 disk.sda.in‚0$ memory_percent‚99.96142387390137 disk.rbd3.out‚0) disk./.usagePercent‚74.37499728632014% disk./.usageGB‚1108.2412071228027! disk.dm-0.out‚5153.4111328125 disk.sda2.in‚0) proc.memory.percent‚17.48049259185791 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 49.5859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0öG¸žÆ‚(:“ Û£æÔá¦ÏÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.806399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]!²Ç‚(: Û£æÔ†€ÑØ disk.drbd1777.out‚0 network.recv‚544292 disk.sda.out‚ 50.515625 disk.sda3.out‚ 50.515625) proc.memory.percent‚17.48049259185791 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚30.43408203125 disk.nvme0n1.in‚ 158.40234375# proc.memory.rssMB‚ 2864.00390625 network.sent‚ 2.812137e+06 disk.dm-1.out‚ 50.515625 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 158.40234375 disk.sda.in‚0! disk.dm-0.out‚5153.8603515625$ memory_percent‚99.96140003204346) disk./.usagePercent‚74.37513117826055 disk.rbd2.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 6.32421875 cpu‚25.298494523820015% disk./.usageGB‚1108.2432022094727 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 158.40234375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚5153.8603515625$ disk.nvme0n1.out‚5153.8603515625 disk.drbd1777.in‚0¼²r2È- _wandbruntime‚1336  _runtime‚1336D#*ÆÉÀ $ _timestamp‚1.788449244927908e+09 _runtime‚1336.482361078 _step‚139 & lr-AdamW/pg1‚0.0001831667416111406 & lr-AdamW/pg2‚0.0001831667416111406 trainer/global_step‚1749Ù¨'±ËÊÅ & lr-AdamW/pg1‚0.0001831667416111406 & lr-AdamW/pg2‚0.0001831667416111406 trainer/global_step‚1749 $ _timestamp‚1.788449244927908e+09 _runtime‚1336.482361078 _step‚139‹È+82Ë- _wandbruntime‚1336  _runtime‚1336® ©Ó'ÌÍO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100955004281344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102412950907392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32888240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99773999480991 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111826856081920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1457272261632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102412276542976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1457928623104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25216376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58104616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884492449287899e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100955004281344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1451909287723 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8353124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23063008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1452563725471 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111826672711168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32868204 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1749 _runtime‚1336.483446914 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25216374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110295617434112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25216376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110295593494016àGÂ;ÓÍÒO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32888240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99773999480991 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111826856081920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1457272261632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102412276542976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1457928623104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25216376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 MGDeviceStatsMonitor.on_train_batch_start/reserved_b·6üq"ytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58104616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884492449287899e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32868204 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100955004281344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1451909287723 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32888240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8353124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23063008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1452563725471 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111826672711168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32868204 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1749 _runtime‚1336.483446914 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25216374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100955022284288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110295617434112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25216376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100955022284288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110295593494016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100955004281344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98321418716160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102412950907392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Œ„íàõ2Î- _wandbruntime‚1337  _runtime‚1337"Þñ¿Ü&ÏÖM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚141 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 _runtime‚1337.193714048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952D>’Öá&ÐÛM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1458748064256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚1749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚141 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98376339886080 _runtime‚1337.193714048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÒÖ2Ñ- _wandbruntime‚1337  _runtime‚1337h?Æ7VÒÐ 1loss/SRqqt1/detection_loss‚1.8185134058197339 1loss/SRqqt2/detection_loss‚1.4831347312364314 epoch‚1 0loss/FRt3/assignment_loss‚4.3246417498588565 1loss/FBt4/assignment_loss‚0.42209690870800487 /loss/FRt1/detection_loss‚2.3735527247190475 $ _timestamp‚1.788449245640591e+09 /loss/FRt2/detection_loss‚1.9338769578645307 /loss/FRt4/detection_loss‚2.9854293992886176 1loss/SRqqt3/detection_loss‚2.0459863287210465 1loss/SRqqt4/detection_loss‚2.5632781934093787 .loss/FBt3/detection_loss‚4.327286710341771 .loss/FBt4/detection_loss‚4.860546151796977 _step‚142 0loss/FRt2/assignment_loss‚3.1107069196239596 /loss/FBt1/detection_loss‚3.3144055803616843 /loss/FBt2/detection_loss‚1.1835532102561537 &loss/total_loss‚57.166919568627605 1loss/SRqqt3/assignment_loss‚1.864124744683504 1loss/SRqqt4/assignment_loss‚2.643129153831585 .loss/FRt3/detection_loss‚2.151084260940552 _runtime‚1337.194942012 2loss/SRqqt1/assignment_loss‚1.8661959665517012 1loss/FBt1/assignment_loss‚0.30685890403735055 /loss/FBt2/assignment_loss‚0.253518434497449 trainer/global_step‚1749 ,loss/FRt1/assignment_loss‚3.572097222507 /loss/FRt4/assignment_loss‚5.621041958148663 1loss/SRqqt2/assignment_loss‚1.667050595395267 1loss/FBt3/assignment_loss‚0.47480935602834506M®v[ÓÕ /loss/FRt2/detection_loss‚1.9338769578645307 /loss/FRt4/detection_loss‚2.9854293992886176 1loss/SRqqt3/detection_loss‚2.0459863287210465 1loss/SRqqt4/detection_loss‚2.5632781934093787 .loss/FBt3/detection_loss‚4.327286710341771 .loss/FBt4/detection_loss‚4.860546151796977 _step‚142 0loss/FRt2/assignment_loss‚3.1107069196239596 /loss/FBt1/detection_loss‚3.3144055803616843 /loss/FBt2/detection_loss‚1.1835532102561537 &loss/total_loss‚57.166919568627605 1loss/SRqqt3/assignment_loss‚1.864124744683504 1loss/SRqqt4/assignment_loss‚2.643129153831585 .loss/FRt3/detection_loss‚2.151084260940552 _runtime‚1337.194942012 2loss/SRqqt1/assignment_loss‚1.8661959665517012 1loss/FBt1/assignment_loss‚0.30685890403735055 /loss/FBt2/assignment_loss‚0.253518434497449 trainer/global_step‚1749 ,loss/FRt1/assignment_loss‚3.572097222507 /loss/FRt4/assignment_loss‚5.621041958148663 1loss/SRqqt2/assignment_loss‚1.667050595395267 1loss/FBt3/assignment_loss‚0.47480935602834506 1loss/SRqqt1/detection_loss‚1.8185134058197339 1loss/SRqqt2/detection_loss‚1.4831347312364314 epoch‚1 0loss/FRt3/assignment_loss‚4.3246417498588565 1loss/FBt4/assignment_loss‚0.42209690870800487 /loss/FRt1/detection_loss‚2.3735527247190475 $ _timestamp‚1.788449245640591e+09Žd{éžÔ‚(:“ ê£æÔÔ°úÕ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.873' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.174599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘#e LÕ‚(:‡ ê£æÔ¾æµÙ) proc.memory.percent‚17.48049259185791 network.recv‚548425 disk.rbd1.out‚ 0.61328125" disk.dm-0.out‚5156.35693359375 disk.sda2.out‚0 network.sent‚ 2.81834e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 158.40625 disk.drbd1777.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 6.3359375 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.out‚ 50.984375 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚30.44970703125 cpu‚25.295754543767757 disk.rbd3.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 158.40625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚5156.35693359375 disk.dm-1.out‚ 50.984375$ memory_percent‚99.96132850646973 disk.sda1.in‚0% disk.nvme0n1.out‚5156.35693359375àitÆ# proc.memory.rssMB‚ 2864.00390625) disk./.usagePercent‚74.37520106831738% disk./.usageGB‚1108.2442436218262 disk.sda.out‚ 50.984375 disk.nvme0n1.in‚ 158.40625 disk.dm-1.in‚0„R>“Ö‚(:ˆ ù£æÔ–ÃÜÕ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.024' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌXjׂ(:Š ù£æÔù´ˆØ) disk./.usagePercent‚74.37534110443862 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 158.40625" disk.dm-0.out‚5156.85595703125 disk.sda2.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 158.40625 disk.sda2.in‚0 disk.dm-1.out‚ 51.8203125$ memory_percent‚99.96178150177002 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda.out‚ 51.8203125 cpu‚25.29315482412087 network.recv‚554419 disk.sda3.in‚0 disk.sda3.out‚ 51.8203125 disk.rbd1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2864.00390625 disk.rbd2.out‚ 32.0947265625 disk.dm-1.in‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚5156.85595703125% disk.nvme0n1.out‚5156.85595703125 disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0 network.sent‚ 2.926218e+06% disk./.usageGB‚1108.2463302612305 disk.nvme0n1.in‚ 158.40625& proc.memory.availableMB‚ 6.26171875) proc.memory.percent‚17.48049259185791ÞX¸2Ø- _wandbruntime‚1374  _runtime‚1374,´ëÇÙÁ & lr-AdamW/pg2‚0.0001829507149033335 trainer/global_step‚1799 % _timestamp‚1.7884492825227017e+09 _runtime‚1374.077052089 _step‚143 & lr-AdamW/pg1‚0.0001829507149033335Ž©M*ÌÚÆ _runtime‚1374.077052089 _step‚143 & lr-AdamW/pg1‚0.0001829507149033335 & lr-AdamW/pg2‚0.0001829507149033335 trainer/global_step‚1799 % _timestamp‚1.7884492825227017e+09ªuÁò2Û- _wandbruntime‚1374  _runtime‚1374;x¾óÖ'ÜÐO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1498258067456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105271588655616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1574072062464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788449282523406e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1498914428928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25917176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59711430 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814292 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101066642283520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 _step‚144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8582033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚1374.077982092 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1492744627131 trainer/global_step‚1799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103772656223744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102560041348399 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8582033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814292 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113378154317312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498258067456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚1374.077982092 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1492744627131 trainer/global_step‚1799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1574233590272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103772656223744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102560041348399 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788449282523406e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1498914428928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25917176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 6743644166ìj-2Þ- _wandbruntime‚1374  _runtime‚1374ÚHá&ßÛM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33832772 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105328761135616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1499733870080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15126022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8586091 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1575093943296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25931192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1374.781122981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101121529374720 trainer/global_step‚1799ÍÏYAƒààM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23712555 TFDeviceStatsMonitor.on_train_batch_end/in§šÜc&active_split_bytes.all.current‚ 195940352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113439781114880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884492832262456e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103829009262592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33812710 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚145 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8586091 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1575093943296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25931192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1374.781122981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101121529374720 trainer/global_step‚1799 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33832772 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105328761135616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1499733870080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15126022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 _runtime‚1411.027799366 _step‚148 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106590308166144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚1849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26617974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884493194731972e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108129552039424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1533579966539 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105346083215807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34740344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539243873280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105345411738699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34720308 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Ð\ô5sïÖO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34740344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539243873280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105345411738699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34720308 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103811848811520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 _runtime‚1411.027799366 _step‚148 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106590308166144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1617179271168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚1849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26617974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039¶fyi360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884493194731972e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539900234752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108129552039424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1533579966539 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105346083215807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088”n h2ð- _wandbruntime‚1411  _runtime‚1411Íjãá&ñÛM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1411.723022104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24349469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116522317998080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 % _timestamp‚1.7884493201680238e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116522341938176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚1849 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _step‚149 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106646661204992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106646679207936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34738762 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15534420 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108187398883840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1540063301120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26631992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0V`›æ&òàM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106646679207936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34738762 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15534420 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118140549621248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108187398883840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1540063301120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26631992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1411.723022104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24349469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116522317998080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 % _timestamp‚1.7884493201680238e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116522341938176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚1849 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _step‚149 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106646661204992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105401132304223•9"2ó- _wandbruntime‚1411  _runtime‚1411à¶OãUôÏ 0loss/FBt1/assignment_loss‚0.8145940133503505 0loss/FBt3/assignment_loss‚0.6610501207558793 1loss/SRqqt1/detection_loss‚1.7547773491528074 /loss/FBt2/detection_loss‚0.6855546264511494 trainer/global_step‚1849 _step‚150 0loss/FBt2/assignment_loss‚0.3450041664959618 /loss/SRqqt2/detection_loss‚1.27900959091422 /loss/FBt1/detection_loss‚2.9635694878441945 /loss/FBt4/detection_loss‚3.9822284314367504 /loss/FRt4/assignment_loss‚4.866259034942178 .loss/FRt4/detection_loss‚2.979154919876772 /loss/FBt3/detection_loss‚2.7460459795865146 2loss/SRqqt3/assignment_loss‚2.1167288531859714 /loss/FRt2/detection_loss‚1.9831730507789773 /loss/FRt3/assignment_loss‚4.238744854927063 0loss/SRqqt3/detection_loss‚2.121733102533552 epoch‚1 _runtime‚1411.724135142 1loss/SRqqt1/assignment_loss‚2.124303759659751 .loss/FRt1/detection_loss‚2.578268891850183 %loss/total_loss‚55.31877735061369 $ _timestamp‚1.788449320169826e+09 0loss/FRt1/assignment_loss‚4.3958719913349595 1loss/FBt4/assignment_loss‚0.43593003931970187 .loss/FRt3/detection_loss‚2.168420441010419 0loss/SRqqt4/detection_loss‚2.785802788204617 0loss/FRt2/assignment_loss‚2.7342958564453936 2loss/SRqqt2/assignment_loss‚1.7664994037261716 2loss/SRqqt4/assignment_loss‚2.7917565968301563œ”ý-ZõÔ 0loss/FRt1/assignment_loss‚4.3958719913349595 1loss/FBt4/assignment_loss‚0.43593003931970187 .loss/FRt3/detection_loss‚2.168420441010419 0loss/SRqqt4/detection_loss‚2.785802788204617 0loss/FRt2/assignment_loss‚2.7342958564453936 2loss/SRqqt2/assignment_loss‚1.7664994037261716 2loss/SRqqt4/assignment_loss‚2.7917565968301563 0loss/FBt1/assignment_loss‚0.8145940133503505 0loss/FBt3/assignment_loss‚0.6610501207558793 1loss/SRqqt1/detection_loss‚1.7547773491528074 /loss/FBt2/detection_loss‚0.6855546264511494 trainer/global_step‚1849 _step‚150 0loss/FBt2/assignment_loss‚0.3450041664959618 /loss/SRqqt2/detection_loss‚1.27900959091422 /loss/FBt1/detection_loss‚2.9635694878441945 /loss/FBt4/detection_loss‚3.9822284314367504 /loss/FRt4/assignment_loss‚4.866259034942178 .loss/FRt4/detection_loss‚2.979154919876772 /loss/FBt3/detection_loss‚2.7460459795865146 2loss/SRqqt3/assignment_loss‚2.1167288531859714 /loss/FRt2/detection_loss‚1.9831730507789773 /loss/FRt3/assignment_loss‚4.238744854927063 0loss/SRqqt3/detection_loss‚2.121733102533552 epoch‚1 _runtime‚1411.724135142 1loss/SRqqt1/assignment_loss‚2.124303759659751 .loss/FRt1/detection_loss‚2.578268891850183 %loss/total_loss‚55.31877735061369 $ _timestamp‚1.788449320169826e+09–>Þ £“ö‚(:ˆ µ¤æÔéë™× gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”n'Á÷‚(:Š µ¤æÔÄÛÆØ disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 158.40625$ disk.nvme0n1.out‚5160.5810546875! disk.dm-0.out‚5160.5810546875 disk.dm-1.in‚0$ proc.memory.availableMB‚5.859375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 network.sent‚ 3.052174e+06 network.recv‚574084% disk./.usageGB‚1108.0768280029297 disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0 disk.rbd2.out‚32.27490234375$ memory_percent‚99.96423721313477 disk.dm-1.out‚ 54.52734375# proc.memory.rssMB‚ 2864.00390625) proc.memory.percent‚17.48049259185791 disk.sda3.out‚ 54.52734375) disk./.usagePercent‚74.36396566566219 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚5160.5810546875 disk.dm-0.in‚ 158.40625 disk.sda2.out‚0 disk.sda.out‚ 54.52734375 disk.rbd2.in‚0 cpu‚25.283252361424942 disk.nvme0n1.in‚ 158.40625 disk.drbd1777.out‚0âI!ûžø‚(:“ ĤæÔܽÚÖ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.868' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.773600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¾3Ìù‚(:„ ĤæÔÀ‹ŸØ disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2864.00390625 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚5288.3037109375 disk.dm-1.out‚ 54.9765625 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚580618 disk.rbd0.in‚0 disk.dm-0.in‚ 158.40625 disk.dm-1.in‚0 cpu‚25.28087621263824 network.sent‚ 3.160728e+06% disk./.usageGB‚1108.2005996704102 disk.sda3.in‚0! disk.dm-0.out‚5288.3037109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 54.9765625) disk./.usagePercent‚74.37227208611816 disk.rbd2.out‚32.29833984375 disk.nvme0n1.in‚ 158.40625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 54.9765625 disk.nvme0c0n1.in‚ 158.40625$ disk.nvme0n1.out‚5288.3037109375 disk.drbd1777.in‚0# proc.memory.availableMB‚6.34375 proc.cpu.threads‚17# memory_percent‚99.9612808227539) proc.memory.percent‚17.48049259185791 disk.sda1.out‚0'l²}2ú- _wandbruntime‚1448  _runtime‚1448íšÇûÁ & lr-AdamW/pg1‚0.0001824619373672557 & lr-AdamW/pg2‚0.0001824619373672557 trainer/global_step‚1899 % _timestamp‚1.7884493569251359e+09 _runtime‚1448.479552344 _step‚151,¿ÌüÆ & lr-AdamW/pg1‚0.0001824619373672557 & lr-AdamW/pg2‚0.0001824619373672557 trainer/global_step‚1899 % _timestamp‚1.7884493569251359e+09 _runtime‚1448.479552344 _step‚151—Ü­Øg2ý- _wandbruntime‚1448  _runtime‚1448ß(½þÐO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110988864152064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119543228083712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDevis“ÐÕ"ceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27318776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35666396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1580229679104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35646360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110988864152064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9039448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62965134 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1899 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15934652 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1580886040576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1899 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15934652 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108131453606107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1580886040576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119543228083712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27318776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35666396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1580229679104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35646360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110988864152064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9039448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62965134 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788449357637438e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108187845648947 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9044029 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27332790 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193843200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27332790 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27332790 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108187174171631 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/s®›œÊegment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193843200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788449357637438e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1575886178867 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569™×Ð2ƒ- _wandbruntime‚1449  _runtime‚1449¯s}º\„Ö 2loss/SRqqt1/assignment_loss‚1.6833035772497005 1loss/FBt2/assignment_loss‚0.29620516549992854 /loss/FRt4/detection_loss‚3.0902569161521063 trainer/global_step‚1899 /loss/FRt1/detection_loss‚2.3237444688665105 /loss/FBt3/detection_loss‚2.6102981811219994 .loss/FBt4/detection_loss‚3.397683253655067 1loss/SRqqt1/detection_loss‚1.8107242421670393 1loss/SRqqt3/detection_loss‚1.8495268014570077 0loss/SRqqt4/detection_loss‚3.157601147890091 0loss/FBt4/assignment_loss‚0.2948925387367686 _step‚154 2loss/SRqqt2/assignment_loss‚1.6065555091311292 1loss/SRqqt3/assignment_loss‚2.055236974110206 1loss/SRqqt2/detection_loss‚1.2569305235292854 epoch‚1 _runtime‚1449.193587378 0loss/FRt4/assignment_loss‚5.0855686399671765 /loss/FRt2/detection_loss‚1.9589708672379547 /loss/FRt3/detection_loss‚2.1637419047562974 /loss/FBt1/detection_loss‚3.4765180945396423 .loss/FBt2/detection_loss‚1.035772674138974 &loss/total_loss‚52.812073337485145 % _timestamp‚1.7884493576392934e+09 0loss/FRt2/assignment_loss‚2.7706807300729572 /loss/FRt3/assignment_loss‚4.241346810175025 1loss/SRqqt4/assignment_loss‚2.200774882150733 1loss/FBt1/assignment_loss‚0.18863281777956636 0loss/FBt3/assignment_loss‚0.3485242924226565 0loss/FRt1/assignment_loss‚3.9085823246773255{îða…Û /loss/FRt4/detection_loss‚3.0902569161521063 trainer/global_step‚1899 /loss/FRt1/detection_loss‚2.3237444688665105 /loss/FBt3/detection_loss‚2.6102981811219994 .loss/FBt4/detection_loss‚3.397683253655067 1loss/SRqqt1/detection_loss‚1.8107242421670393 1loss/SRqqt3/detection_loss‚1.8495268014570077 0loss/SRqqt4/detection_loss‚3.157601147890091 0loss/FBt4/assignment_loss‚0.2948925387367686 _step‚154 2loss/SRqqt2/assignment_loss‚1.6065555091311292 1loss/SRqqt3/assignment_loss‚2.055236974110206 1loss/SRqqt2/detection_loss‚1.2569305235292854 epoch‚1 _runtime‚1449.193587378 0loss/FRt4/assignment_loss‚5.0855686399671765 /loss/FRt2/detection_loss‚1.9589708672379547 /loss/FRt3/detection_loss‚2.1637419047562974 /loss/FBt1/detection_loss‚3.4765180945396423 .loss/FBt2/detection_loss‚1.035772674138974 &loss/total_loss‚52.812073337485145 % _timestamp‚1.7884493576392934e+09 0loss/FRt2/assignment_loss‚2.7706807300729572 /loss/FRt3/assignment_loss‚4.241346810175025 1loss/SRqqt4/assignment_loss‚2.200774882150733 1loss/FBt1/assignment_loss‚0.18863281777956636 0loss/FBt3/assignment_loss‚0.3485242924226565 0loss/FRt1/assignment_loss‚3.9085823246773255 2loss/SRqqt1/assignment_loss‚1.6833035772497005 1loss/FBt2/assignment_loss‚0.29620516549992854šÃ2{𓆂(:ˆ Ó¤æÔ¦ÈìÖ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.368' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûý燂(:‡ Ó¤æÔŽí¸Ø proc.cpu.threads‚17% disk./.usageGB‚1108.3241996765137 disk.rbd3.out‚0 disk.sda.in‚0& disk.nvme0c0n1.out‚5415.2685546875 disk.drbd1777.out‚0! disk.dm-0.out‚5415.2685546875 disk.sda1.in‚0 network.sent‚ 3.165256e+06) disk./.usagePercent‚74.38056698623507 disk.nvme0n1.in‚ 158.40625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda3.in‚0" proc.memory.availableMB‚6.0625 cpu‚25.27869925704291 disk.rbd1.in‚0 disk.sda3.out‚ 55.68359375# proc.memory.rssMB‚ 2864.00390625 disk.sda2.in‚0 network.recv‚583852$ disk.nvme0n1.out‚5415.2685546875$ memory_percent‚99.96299743652344 disk.dm-1.out‚ 55.68359375 disk.rbd3.in‚0 disk.dm-0.in‚ 158.40625 disk.sda.out‚ 55.68359375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.rbd2.out‚33.95068359375 disk.nvme0c0n1.in‚ 158.40625) proc.memory.percent‚17.48049259185791{¢;ûžˆ‚(:“ â¤æÔ®¬±× gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.858400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êb‰‚(:Š â¤æÔ‚ªäØ disk.sda3.in‚0 disk.nvme0c0n1.in‚ 158.40625 proc.cpu.threads‚17& disk.nvme0c0n1.out‚5416.2939453125 disk.drbd1777.in‚0! disk.dm-0.out‚5416.2939453125 network.sent‚ 3.273888e+06 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 56.16796875 disk.rbd0.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.38057108235562 disk.sda2.in‚0 disk.dm-1.out‚ 56.16796875 disk.sda1.out‚0 disk.dm-0.in‚ 158.40625& proc.memory.availableMB‚ 5.66015625 disk.sda1.in‚0 disk.sda3.out‚ 56.16796875 disk.rbd2.out‚33.96240234375$ memory_percent‚99.96545314788818 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.48049259185791 disk.nvme0n1.in‚ 158.40625$ disk.nvme0n1.out‚5416.2939453125# proc.memory.rssMB‚ 2864.00390625 cpu‚25.276560805660612 disk.rbd1.out‚ 0.61328125 network.recv‚590419# disk./.usageGB‚1108.32426071167 disk.sda2.out‚0ÚâÂU“Š‚(:ˆ ñ¤æÔ¤Ûî× gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œû5†‹‚(:ƒ ñ¤æÔÆ¥ÆØ) proc.memory.percent‚17.48049259185791 disk.sda1.out‚0 disk.nvme0n1.in‚ 158.40625$ disk.nvme0n1.out‚5416.5361328125 disk.dm-0.in‚ 158.40625 disk.rbd1.out‚ 0.61328125 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 158.40625 disk.sda2.in‚0 disk.dm-1.out‚ 56.8828125 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.38057108235562 disk.sda.in‚0 disk.sda3.out‚ 56.8828125! disk.dm-0.out‚5416.5361328125# proc.memory.rssMB‚ 2864.00390625 cpu‚25.274153708540375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 34.048828125 disk.sda.out‚ 56.8828125 disk.rbd0.out‚0# disk./.usageGB‚1108.32426071167 disk.sda1.in‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚5416.5361328125 disk.drbd1777.in‚0 disk.rbd1.in‚0$ memory_percent‚99.96557235717773 network.sent‚ 3.279472e+06 network.recv‚594663 disk.rbd3.in‚0$ proc.memory.availableMB‚5.640625#5R32Œ- _wandbruntime‚1488  _runtime‚1488ÚÁÙÉà trainer/global_step‚1949 % _timestamp‚1.7884493970672204e+09 _runtime‚1488.621600615 _step‚155 ' lr-AdamW/pg1‚0.00018218928839949092 ' lr-AdamW/pg2‚0.00018218928839949092 1ŽäÎŽÈ ' lr-AdamW/pg1‚0.00018218928839949092 ' lr-AdamW/pg2‚0.00018218928839949092 trainer/global_step‚1949 % _timestamp‚1.7884493970672204e+09 _runtime‚1488.621600615 _step‚155›WBËy2- _wandbruntime‚1488  _runtime‚1488FÓ'×'ÑO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16343052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615250645355 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110917495473515 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109302244828160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124329221809664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36592448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109302261867520 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1949 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112225630053888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615905083103 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _runtime‚1488.622666995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884493970680072e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615905083103 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _runtime‚1488.622666995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884493970680072e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110917495473515 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109302244828160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124329221809664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36592448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109302261867520 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_bat£*%ãÐch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1949 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112225630053888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844œxaX2’- _wandbruntime‚1489  _runtime‚1489)“Êïá&“ÛM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28033592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚1949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36610928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28033590 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124391559815680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124391738978816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1704168051200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64624456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1489.326638929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚157»Âß\æ&”àM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616067080319 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179163136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122687415704576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28033592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616721518275 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚1949 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36610928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28033590 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124391559815680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124391738978816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1704168051200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64624456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚1489.326638929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 FN2œ- _wandbruntime‚1525  _runtime‚1525:"¿È $ _timestamp‚1.788449434080368e+09 _runtime‚1525.634810768 _step‚159 ' lr-AdamW/pg1‚0.00018189787375056356 ' lr-AdamW/pg2‚0.00018189787375056356 trainer/global_step‚1999õBåÍžÇ trainer/global_step‚1999 $ _timestamp‚1.788449434080368e+09 _runtime‚1525.634810768 _step‚159 ' lr-AdamW/pg1‚0.00018189787375056356 ' lr-AdamW/pg2‚0.00018189787375056356ŸL §2Ÿ- _wandbruntime‚1525  _runtime‚1525„ÀóàÐO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚1525.635789142 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518500 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1662857652224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceS ÷$ ItatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1662857652224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113704208818031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 $ _timestamp‚1.788449434081158e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26250249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37498464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125708301850112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127454923034112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1656085984763 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1662201290752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ë%fÛ'¡ÕO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26250249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37498464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125708301850112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28720374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127454923034112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1656085984763 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28720376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1662201290752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚1525.635789142 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518500 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1746621184000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1662857652224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1662857652224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113704208818031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 $ _timestamp‚1.788449434081158e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 x€ú>2¢- _wandbruntime‚1526  _runtime‚1526Ó½á&£ÛM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 _step‚161 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1747313533440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1663677093376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127517242181120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9502929 % _timestamp‚1.7884494347718585e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1663677093376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚1526.326846955 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115099635035136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37516918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980.˜£d¦¤àM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9502929 % _timestamp‚1.7884494347718585e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37516918 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1663677093376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚1526.326846955 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115099635035136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37516918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37536980 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 _step‚161 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 /Eà@ E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1747313533440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚1999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115099617032192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1663677093376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127517242181120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944¡ƒA$2¥- _wandbruntime‚1526  _runtime‚1526æhˆS¦Í /loss/FRt2/assignment_loss‚2.275675182896001 2loss/SRqqt4/assignment_loss‚2.3669909381866456 0loss/FBt4/assignment_loss‚0.5854442565664775 0loss/SRqqt1/detection_loss‚2.167433652073838 /loss/SRqqt4/detection_loss‚2.80285902261734 /loss/FRt1/assignment_loss‚4.685360525593613 1loss/SRqqt3/assignment_loss‚1.714010555635799 /loss/FRt2/detection_loss‚1.9998407631979458 .loss/FRt4/detection_loss‚3.139673058803265 /loss/FBt2/detection_loss‚0.6736264781911155 epoch‚1 0loss/FRt3/assignment_loss‚3.8071801960468292 2loss/SRqqt1/assignment_loss‚2.2514262830102165 0loss/FBt2/assignment_loss‚0.1872711519840493 1loss/SRqqt2/detection_loss‚1.3721785330671374 0loss/SRqqt3/detection_loss‚2.041030988943848 _step‚162 1loss/FBt1/assignment_loss‚0.42858812771737576 0loss/FBt3/assignment_loss‚0.5605450323781724 .loss/FRt3/detection_loss‚2.289166682440302 /loss/FBt3/detection_loss‚2.4116288331838756 .loss/FBt4/detection_loss‚3.180351836340768 %loss/total_loss‚53.49485123554173 _runtime‚1526.328226796 .loss/FRt1/detection_loss‚2.809480793548353 .loss/FRt4/assignment_loss‚4.48268433717581 2loss/SRqqt2/assignment_loss‚1.3134830109459363 /loss/FBt1/detection_loss‚3.9489209949970245 trainer/global_step‚1999 % _timestamp‚1.7884494347736368e+09½©CX§Ò /loss/FRt2/assignment_loss‚2.275675182896001 2loss/SRqqt4/assignment_loss‚2.3669909381866456 0loss/FBt4/assignment_loss‚0.5854442565664775 0loss/SRqqt1/detection_loss‚2.167433652073838 /loss/SRqqt4/detection_loss‚2.80285902261734 /loss/FRt1/assignment_loss‚4.685360525593613 1loss/SRqqt3/assignment_loss‚1.714010555635799 /loss/FRt2/detection_loss‚1.9998407631979458 .loss/FRt4/detection_loss‚3.139673058803265 /loss/FBt2/detection_loss‚0.6736264781911155 epoch‚1 0loss/FRt3/assignment_loss‚3.8071801960468292 2loss/SRqqt1/assignment_loss‚2.2514262830102165 0loss/FBt2/assignment_loss‚0.1872711519840493 1loss/SRqqt2/detection_loss‚1.3721785330671374 0loss/SRqqt3/detection_loss‚2.041030988943848 _step‚162 1loss/FBt1/assignment_loss‚0.42858812771737576 0loss/FBt3/assignment_loss‚0.5605450323781724 .loss/FRt3/detection_loss‚2.289166682440302 /loss/FBt3/detection_loss‚2.4116288331838756 .loss/FBt4/detection_loss‚3.180351836340768 %loss/total_loss‚53.49485123554173 _runtime‚1526.328226796 .loss/FRt1/detection_loss‚2.809480793548353 .loss/FRt4/assignment_loss‚4.48268433717581 2loss/SRqqt2/assignment_loss‚1.3134830109459363 /loss/FBt1/detection_loss‚3.9489209949970245 trainer/global_step‚1999 % _timestamp‚1.7884494347736368e+09¢öÏÝ8ž¨‚(:“ ž¥æÔž©ñÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.423000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&C5©‚(:Œ ž¥æÔõÙûØ disk.sda.out‚ 58.36328125 disk.rbd2.in‚0 network.recv‚609747% disk./.usageGB‚1108.3328590393066 disk.dm-0.in‚ 158.40625& proc.memory.availableMB‚ 5.37890625 cpu‚25.267816062397504 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 158.40625 network.sent‚ 3.399485e+06 disk.rbd1.out‚ 0.61328125' disk.nvme0c0n1.out‚5426.54541015625$ memory_percent‚99.96716976165771 disk.dm-1.out‚ 58.36328125 disk.rbd0.out‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2864.00390625) proc.memory.percent‚17.48049259185791 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚0" disk.dm-0.out‚5426.54541015625% disk.nvme0n1.out‚5426.54541015625( disk./.usagePercent‚74.3811481233378 disk.sda3.out‚ 58.36328125 disk.rbd0.in‚0 disk.rbd2.out‚ 34.142578125 disk.nvme0c0n1.in‚ 158.40625 disk.rbd1.in‚0 disk.rbd3.in‚0°©¿“ª‚(:ˆ ­¥æÔô¬ôÕ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÈäK«‚(:Œ ­¥æÔ„¦ÚØ disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 58.94921875 disk.rbd3.out‚0 disk.sda.out‚ 58.94921875 disk.sda3.in‚0) proc.memory.percent‚17.48049259185791 disk.sda.in‚0 disk.rbd2.out‚ 35.779296875 disk.sda1.out‚0% disk.nvme0n1.out‚5426.88525390625$ memory_percent‚99.96683597564697# proc.memory.rssMB‚ 2864.00390625 disk.sda3.out‚ 58.94921875 disk.drbd1777.in‚0 network.recv‚615856% disk./.usageGB‚1108.3328590393066 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 158.40625" disk.dm-0.out‚5426.88525390625 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚5426.88525390625 disk.dm-1.in‚0& proc.memory.availableMB‚ 5.43359375 network.sent‚ 3.507522e+06 disk.rbd1.out‚ 0.61328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 158.40625 disk.dm-0.in‚ 158.40625 cpu‚25.265585672720196 proc.cpu.threads‚17( disk./.usagePercent‚74.3811481233378Ž¼Xž¬‚(:“ ¼¥æÔäç½Õ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.423000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡±­‚(:‹ ¼¥æÔÄÒÒÙ disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚5427.39697265625 disk.sda3.out‚ 59.70703125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2864.01171875 disk.nvme0c0n1.in‚ 158.40625 disk.nvme0n1.in‚ 158.40625% disk.nvme0n1.out‚5427.39697265625$ disk./.usageGB‚1108.332462310791 disk.rbd2.out‚ 35.794921875$ memory_percent‚99.96762275695801% proc.memory.availableMB‚ 5.3046875 network.sent‚ 3.513971e+06 disk.rbd1.out‚ 0.76171875 disk.sda1.out‚0 disk.sda3.in‚0 cpu‚25.263673045131895 proc.cpu.threads‚17) disk./.usagePercent‚74.38112149855424 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 59.70703125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 158.40625 disk.dm-1.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚5427.39697265625 disk.rbd1.in‚0 disk.sda.out‚ 59.70703125) proc.memory.percent‚17.48054027557373 network.recv‚620580ƾ›â2®- _wandbruntime‚1563  _runtime‚1563Cìxɯà ' lr-AdamW/pg1‚0.00018158775415084966 ' lr-AdamW/pg2‚0.00018158775415084966 trainer/global_step‚2049 % _timestamp‚1.7884494715488791e+09 _runtime‚1563.103393015 _step‚163 á]±Î°È _step‚163 ' lr-AdamW/pg1‚0.00018158775415084966 ' lr-AdamW/pg2‚0.00018158775415084966 trainer/global_step‚2049 % _timestamp‚1.7884494715488791e+09 _runtime‚1563.103393015£^ΩÁ2±- _wandbruntime‚1563  _runtime‚1563+e^$×'²ÑO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177079296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117860933938688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _step‚164 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26888238 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1703187096576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38424516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789577584128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789730723328 _runtime‚1563.104411194 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128790814793216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119564777396736szGÜ'³ÖO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119564777396736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67865728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117860915935744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177079296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117860933938688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116490250685439 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 _step‚164 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26888238 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1703187096576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38424516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1789577584128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1789730723328 _runtime‚1563.104411194 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128790814793216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928¤ÂW2´- _wandbruntime‚1563  _runtime‚1563çòÿrµÛM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1704662899200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26900854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚165 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38463032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790423271424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697737759135 J;DeviceStatsMonitor.on_train_b©=Ío atch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29435192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117917268974592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17168022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130643084742656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1704662899200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17168020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884494722330368e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452˜4\æ&¶àM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17168022 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130643084742656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1704662899200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17168020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884494722330368e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1704662899200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26900854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚165 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38463032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1790595271680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1790423271424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114847579054080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1697737759135 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29435192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117917268974592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728¥¶ ›T2·- _wandbruntime‚1563  _runtime‚1563È{ÝFU¸Ï 0loss/FRt2/assignment_loss‚2.7830293075396466 /loss/FRt4/assignment_loss‚4.936448008925827 0loss/FBt1/assignment_loss‚0.5669957682490349 1loss/SRqqt3/detection_loss‚2.2882076737127806 /loss/FRt1/assignment_loss‚3.340803789821538 1loss/SRqqt1/assignment_loss‚2.051882569820194 0loss/FBt4/assignment_loss‚0.3339080024445593 /loss/FRt3/detection_loss‚2.0851204039757714 % _timestamp‚1.7884494722352936e+09 /loss/FRt3/assignment_loss‚4.233079199205365 1loss/SRqqt2/assignment_loss‚1.401014275216269 0loss/FBt3/assignment_loss‚0.6139135999893597 .loss/FRt4/detection_loss‚3.166006565093994 _runtime‚ 1563.78957402 .loss/FBt4/detection_loss‚4.185781233012676 epoch‚1 trainer/global_step‚2049 1loss/SRqqt1/detection_loss‚1.8050099636538555 1loss/SRqqt2/detection_loss‚1.4415081532074596 2loss/SRqqt4/assignment_loss‚2.2082937786291383 /loss/FRt1/detection_loss‚2.3887057866562498 /loss/FRt2/detection_loss‚1.9616966041234822 .loss/FBt1/detection_loss‚2.331779587268829 _step‚166 /loss/FBt3/detection_loss‚2.8846051156520844 %loss/total_loss‚52.64546270644196 2loss/SRqqt3/assignment_loss‚1.9902816442282576 0loss/FBt2/assignment_loss‚0.1746653598821667 0loss/SRqqt4/detection_loss‚2.627880676039334 /loss/FBt2/detection_loss‚0.8448456400940858ÿAZ¹Ô /loss/FBt3/detection_loss‚2.8846051156520844 %loss/total_loss‚52.64546270644196 2loss/SRqqt3/assignment_loss‚1.9902816442282576 0loss/FBt2/assignment_loss‚0.1746653598821667 0loss/SRqqt4/detection_loss‚2.627880676039334 /loss/FBt2/detection_loss‚0.8448456400940858 0loss/FRt2/assignment_loss‚2.7830293075396466 /loss/FRt4/assignment_loss‚4.936448008925827 0loss/FBt1/assignment_loss‚0.5669957682490349 1loss/SRqqt3/detection_loss‚2.2882076737127806 /loss/FRt1/assignment_loss‚3.340803789821538 1loss/SRqqt1/assignment_loss‚2.051882569820194 0loss/FBt4/assignment_loss‚0.3339080024445593 /loss/FRt3/detection_loss‚2.0851204039757714 % _timestamp‚1.7884494722352936e+09 /loss/FRt3/assignment_loss‚4.233079199205365 1loss/SRqqt2/assignment_loss‚1.401014275216269 0loss/FBt3/assignment_loss‚0.6139135999893597 .loss/FRt4/detection_loss‚3.166006565093994 _runtime‚ 1563.78957402 .loss/FBt4/detection_loss‚4.185781233012676 epoch‚1 trainer/global_step‚2049 1loss/SRqqt1/detection_loss‚1.8050099636538555 1loss/SRqqt2/detection_loss‚1.4415081532074596 2loss/SRqqt4/assignment_loss‚2.2082937786291383 /loss/FRt1/detection_loss‚2.3887057866562498 /loss/FRt2/detection_loss‚1.9616966041234822 .loss/FBt1/detection_loss‚2.331779587268829 _step‚166¦\ö’º‚(:‡ Ë¥æÔ÷¦³Ö gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚97.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M›ÎŠ»‚(: Ë¥æÔÂ×Ø disk.rbd2.in‚0" disk.dm-0.out‚5524.87744140625 disk.sda2.in‚0 disk.sda.out‚ 60.11328125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 0.76171875 disk.drbd1777.out‚0 cpu‚25.261702104794427% disk./.usageGB‚1108.4578590393066 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 158.40625' disk.nvme0c0n1.out‚5524.87744140625 network.recv‚624076 disk.nvme0c0n1.in‚ 158.40625# proc.memory.rssMB‚ 2864.01171875 proc.cpu.threads‚17% disk.nvme0n1.out‚5524.87744140625 disk.dm-1.in‚0$ memory_percent‚99.96645450592041) proc.memory.percent‚17.48054027557373 disk.sda1.out‚0 network.sent‚ 3.519034e+06) disk./.usagePercent‚74.38953697821978 disk.rbd0.in‚0 disk.rbd2.out‚ 35.857421875 disk.nvme0n1.in‚ 158.40625 disk.dm-1.out‚ 60.11328125& proc.memory.availableMB‚ 5.49609375 disk.sda3.out‚ 60.11328125SWnÄž¼‚(:“ Ú¥æÔòé©Ö gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.897' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.179399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶í&N½‚(:† Ú¥æÔ¡Ó“Ø disk.sda2.out‚0 disk.dm-1.out‚ 60.9453125 disk.rbd3.out‚0$ memory_percent‚99.96731281280518) proc.memory.percent‚17.48056411743164 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.7939453125 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 158.40625! disk.dm-0.out‚5525.9794921875 disk.sda2.in‚0 disk.rbd1.in‚0 network.recv‚630323 cpu‚25.25975396835809 proc.cpu.threads‚17 network.sent‚ 3.62746e+06$ disk./.usageGB‚1108.427822113037 disk.rbd3.in‚0 disk.sda.out‚ 60.9453125! proc.memory.rssMB‚ 2864.015625 disk.sda1.out‚0 disk.drbd1777.out‚0) disk./.usagePercent‚74.38752117489517 disk.rbd2.out‚35.95947265625& disk.nvme0c0n1.out‚5525.9794921875 disk.nvme0n1.in‚ 158.40625 disk.sda.in‚0& proc.memory.availableMB‚ 5.35546875 disk.sda3.out‚ 60.9453125 disk.nvme0c0n1.in‚ 158.40625$ disk.nvme0n1.out‚5525.9794921875 disk.drbd1777.in‚0’ÏÝ×2¾- _wandbruntime‚1600  _runtime‚1600á®lÅ¿¿ % lr-AdamW/pg1‚0.000181258994228809 % lr-AdamW/pg2‚0.000181258994228809 trainer/global_step‚2099 % _timestamp‚1.7884495085733461e+09 _runtime‚1600.127721969 _step‚167i3ÂõÊÀÄ _step‚167 % lr-AdamW/pg1‚0.000181258994228809 % lr-AdamW/pg2‚0.000181258994228809 trainer/global_step‚2099 % _timestamp‚1.7884495085733461e+09 _runtime‚1600.127721969§†z2Á- _wandbruntime‚1600  _runtime‚1600ZF¸b ÂÐO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120678585881088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/oversize_sâ²Enºegments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133706048631808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120678585881088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1832864774656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30121974 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚168 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69525076 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30135990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120734920916992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1600.812917983 _step‚169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133768559188992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27539128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884495092576497e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120734938919936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ÛMVæ&ÆàM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120734920916992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1600.812917983 _step‚169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131935026354176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9962706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133768559188992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27539128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884495092576497e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120734938919936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39369022 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69505012 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69525076 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30135990©n”¿„2Ç- _wandbruntime‚1600  _runtime‚1600HÝ=ÈÏ 2loss/SRqqt1/assignment_loss‚1.6210838313050129 1loss/FBt2/assignment_loss‚0.29610917870424686 1loss/SRqqt4/detection_loss‚3.1752051581507144 /loss/FBt3/detection_loss‚2.2072493519101823 0loss/FBt1/assignment_loss‚0.8087872539957365 1loss/SRqqt2/detection_loss‚1.5465300738811494 &loss/total_loss‚53.192829469078674 0loss/FRt2/assignment_loss‚2.3952799659708273 epoch‚1 % _timestamp‚1.7884495092601724e+09 0loss/FRt1/assignment_loss‚3.1361690759658813 /loss/FRt3/assignment_ÐæcFloss‚4.451866418123245 0loss/SRqqt2/assignment_loss‚2.08967094773596 2loss/SRqqt3/assignment_loss‚1.8040984799154103 .loss/FRt4/detection_loss‚3.152229658983372 /loss/FBt1/detection_loss‚2.8398061990737915 .loss/FBt2/detection_loss‚0.800625721609702 /loss/FRt4/assignment_loss‚5.459386410536589 /loss/FRt2/detection_loss‚2.1897959171429924 -loss/FBt4/detection_loss‚3.05927637219429 _runtime‚1600.814494895 0loss/FBt4/assignment_loss‚0.5451325609907209 0loss/SRqqt1/detection_loss‚1.828712992808398 1loss/SRqqt3/detection_loss‚1.9281582130740087 .loss/FRt1/detection_loss‚2.434765402050245 .loss/FRt3/detection_loss‚2.141374864454927 _step‚170 trainer/global_step‚2099 1loss/SRqqt4/assignment_loss‚2.696614843347798 0loss/FBt3/assignment_loss‚0.5849005771534702Ò‹–gZÉÔ 0loss/FBt1/assignment_loss‚0.8087872539957365 1loss/SRqqt2/detection_loss‚1.5465300738811494 &loss/total_loss‚53.192829469078674 0loss/FRt2/assignment_loss‚2.3952799659708273 epoch‚1 % _timestamp‚1.7884495092601724e+09 0loss/FRt1/assignment_loss‚3.1361690759658813 /loss/FRt3/assignment_loss‚4.451866418123245 0loss/SRqqt2/assignment_loss‚2.08967094773596 2loss/SRqqt3/assignment_loss‚1.8040984799154103 .loss/FRt4/detection_loss‚3.152229658983372 /loss/FBt1/detection_loss‚2.8398061990737915 .loss/FBt2/detection_loss‚0.800625721609702 /loss/FRt4/assignment_loss‚5.459386410536589 /loss/FRt2/detection_loss‚2.1897959171429924 -loss/FBt4/detection_loss‚3.05927637219429 _runtime‚1600.814494895 0loss/FBt4/assignment_loss‚0.5451325609907209 0loss/SRqqt1/detection_loss‚1.828712992808398 1loss/SRqqt3/detection_loss‚1.9281582130740087 .loss/FRt1/detection_loss‚2.434765402050245 .loss/FRt3/detection_loss‚2.141374864454927 _step‚170 trainer/global_step‚2099 1loss/SRqqt4/assignment_loss‚2.696614843347798 0loss/FBt3/assignment_loss‚0.5849005771534702 2loss/SRqqt1/assignment_loss‚1.6210838313050129 1loss/FBt2/assignment_loss‚0.29610917870424686 1loss/SRqqt4/detection_loss‚3.1752051581507144 /loss/FBt3/detection_loss‚2.2072493519101823ª³ð ä‘Ê‚(:† é¥æÔ„ŠþÕ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ä³Ë‚(:… é¥æÔ–†éØ cpu‚25.257955940676737% disk./.usageGB‚1108.5532836914062 disk.dm-0.in‚ 158.40625! disk.dm-0.out‚5526.2646484375 disk.rbd1.in‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 5.22265625) proc.memory.percent‚17.48056411743164 network.sent‚ 3.633006e+06 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 158.40625& disk.nvme0c0n1.out‚5526.2646484375 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚35.95947265625 disk.sda2.out‚0 disk.rbd1.out‚ 0.7939453125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 158.40625 disk.dm-1.out‚ 61.421875 disk.sda.in‚0 disk.sda3.out‚ 61.421875 disk.rbd0.out‚0$ disk.nvme0n1.out‚5526.2646484375 disk.drbd1777.in‚0$ memory_percent‚99.96812343597412! proc.memory.rssMB‚ 2864.015625 disk.rbd3.in‚0 disk.sda.out‚ 61.421875 proc.cpu.threads‚17 network.recv‚634771( disk./.usagePercent‚74.3959410066888 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0’ûžÌ‚(:“ ø¥æÔ¶üÊÕ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.908999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ºŸÍ‚(:‹ ø¥æÔ„’»Ù disk.sda2.in‚0 disk.rbd3.out‚0% proc.memory.availableMB‚ 5.3359375! proc.memory.rssMB‚ 2864.015625 disk.dm-0.in‚ 158.40625 disk.rbd2.in‚0& disk.nvme0c0n1.out‚5624.3408203125 disk.nvme0n1.in‚ 158.40625 disk.rbd0.out‚0$ memory_percent‚99.96743202209473) disk./.usagePercent‚74.39587111663195 disk.dm-1.in‚0 disk.rbd1.out‚ 0.85693359375 disk.sda3.out‚ 62.15234375 disk.nvme0c0n1.in‚ 158.40625 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚37.61181640625) proc.memory.percent‚17.48056411743164 disk.dm-1.out‚ 62.15234375 disk.sda.in‚0 disk.sda.out‚ 62.15234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 cpu‚25.25619204520445 proc.cpu.threads‚17% disk./.usageGB‚1108.5522422790527 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 network.sent‚ 3.741459e+06 network.recv‚640882! disk.dm-0.out‚5624.3408203125 disk.sda2.out‚0$ disk.nvme0n1.out‚5624.3408203125\T˜$“΂(:ˆ ‡¦æÔ²åÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ki¬Ï‚(:Š ‡¦æÔïô†Ø$ memory_percent‚99.96812343597412! proc.memory.rssMB‚ 2864.015625 disk.drbd1777.in‚0" disk.dm-0.out‚5624.79150390625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚5624.79150390625 cpu‚25.254223594476088 disk.drbd1777.out‚0 disk.sda.in‚0& proc.memory.availableMB‚ 5.22265625 network.recv‚644844 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 158.40625' disk.nvme0c0n1.out‚5624.79150390625 disk.dm-0.in‚ 158.40625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 62.921875) proc.memory.percent‚17.48056411743164% disk./.usageGB‚1108.6777610778809 disk.dm-1.out‚ 62.921875 disk.sda1.in‚0 disk.nvme0n1.in‚ 158.40625 proc.cpu.threads‚17 network.sent‚ 3.746618e+06) disk./.usagePercent‚74.40429478853859 disk.rbd1.out‚ 0.85693359375 disk.sda3.in‚0 disk.sda3.out‚ 62.921875 disk.rbd2.out‚37.61181640625øky)2Ð- _wandbruntime‚1637  _runtime‚1637eÝ"ÉÑà % _timestamp‚1.7884495457170842e+09 _runtime‚1637.271489467 _step‚171 ' lr-AdamW/pg1‚0.00018091166249751653 ' lr-AdamW/pg2‚0.00018091166249751653 trainer/global_step‚2149Šø±ÎÒÈ _step‚171 ' lr-AdamW/pg1‚0.00018091166249751653 ' lr-AdamW/pg2‚0.00018091166249751653 trainer/global_step‚2149 % _timestamp‚1.7884495457170842e+09 _runtime‚1637.271489467«Œàc2Ó- _wandbruntime‚1637  _runtime‚1637ºùíôØ'ÔÒO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10187812 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120283070940160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30822774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71119432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120283087979520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _runtime‚1637.272529507 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1875868903936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134955888559616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚2149 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122061662943147 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788449545717885e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136831949222912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125282052893184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚172 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40296656 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876036723200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30822774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71119432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120283087979520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _runtime‚1637.272529507 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1875868903936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134955888559616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚2149 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122061662943147 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788449545717885e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136831949222912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125282052893184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 _step‚172 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40296656 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876036723200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785978136064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125339225373184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40295074 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876708381696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_po»QÒol.peak‚24126 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1779408437951 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2149 _runtime‚1637.965801168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125339225373184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120337975070720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884495464106169e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30836790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928`è&ØâM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125339225373184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40295074 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1876708381696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚173 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30836792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1779408437951 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2149 _runtime‚1637.965801168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125339225373184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120337975070720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123552590862336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884495464106169e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30836790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785978136064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71131864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392­õ)Æ2Ù- _wandbruntime‚1637  _runtime‚1637ATEµVÚÐ 1loss/SRqqt1/assignment_loss‚1.893482506275177 0loss/FBt2/assignment_loss‚0.3345238849662275 /loss/FRt4/detection_loss‚3.3986532904885034 _runtime‚1637.967196748 /loss/FRt4/assignment_loss‚4.961212851784446 .loss/FBt3/detection_loss‚2.399936632676558 &loss/total_loss‚51.585886491483194 /loss/FRt1/assignment_loss‚3.669494179578928 2loss/SRqqt2/assignment_loss‚1.4175600060552824 /loss/FRt3/detection_loss‚2.2399063060681024 1loss/SRqqt2/detection_loss‚1.2252085700631141 /loss/FRt2/assignment_loss‚2.521683795452118 .loss/FRt2/detection_loss‚1.872248097062111 1loss/SRqqt4/detection_loss‚2.8118219785392284 0loss/FBt4/assignment_loss‚0.5873238793150887 .loss/FRt3/assignment_loss‚4.05182621628046 1loss/FBt1/assignment_loss‚0.39564981151904377 0loss/FBt3/assignment_loss‚0.6078958128663626 1loss/SRqqt3/detection_loss‚1.7817799232659801 /loss/FBt2/detection_loss‚0.7837781011495577 trainer/global_step‚2149 % _timestamp‚1.7884495464127963e+09 1loss/SRqqt3/assignment_loss‚1.880860859347928 1loss/SRqqt1/detection_loss‚1.9745444267988206 /loss/FBt4/detection_loss‚2.7985830943692815 epoch‚1 _step‚174 1loss/SRqqt4/assignment_loss‚2.308505406603217 .loss/FRt1/detection_loss‚2.482669153274634 /loss/FBt1/detection_loss‚3.18673770768301835²^[ÛÕ trainer/global_step‚2149 % _timestamp‚1.7884495464127963e+09 1loss/SRqqt3/assignment_loss‚1.880860859347928 1loss/SRqqt1/detection_loss‚1.9745444267988206 /loss/FBt4/detection_loss‚2.7985830943692815 epoch‚1 _step‚174 1loss/SRqqt4/assignment_loss‚2.308505406603217 .loss/FRt1/detection_loss‚2.482669153274634 /loss/FBt1/detection_loss‚3.1867377076830183 1loss/SRqqt1/assignment_loss‚1.893482506275177 0loss/FBt2/assignment_loss‚0.3345238849662275 /loss/FRt4/detection_loss‚3.3986532904885034 _runtime‚1637.967196748 /loss/FRt4/assignment_loss‚4.961212851784446 .loss/FBt3/detection_loss‚2.399936632676558 &loss/total_loss‚51.585886491483194 /loss/FRt1/assignment_loss‚3.669494179578928 2loss/SRqqt2/assignment_loss‚1.4175600060552824 /loss/FRt3/detection_loss‚2.2399063060681024 1loss/SRqqt2/detection_loss‚1.2252085700631141 /loss/FRt2/assignment_loss‚2.521683795452118 .loss/FRt2/detection_loss‚1.872248097062111 1loss/SRqqt4/detection_loss‚2.8118219785392284 0loss/FBt4/assignment_loss‚0.5873238793150887 .loss/FRt3/assignment_loss‚4.05182621628046 1loss/FBt1/assignment_loss‚0.39564981151904377 0loss/FBt3/assignment_loss‚0.6078958128663626 1loss/SRqqt3/detection_loss‚1.7817799232659801 /loss/FBt2/detection_loss‚0.7837781011495577®¨iÜ0žÜ‚(:“ –¦æÔ¾×ìÕ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.310599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸHXÏÝ‚(:‹ –¦æÔÞâêØ disk.sda.out‚ 63.48046875 disk.sda2.out‚0 disk.rbd1.out‚ 0.919921875) proc.memory.percent‚17.48056411743164' disk.nvme0c0n1.out‚5722.93701171875% disk.nvme0n1.out‚5722.93701171875 disk.dm-1.in‚0 network.sent‚ 3.752857e+06 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚5722.93701171875& proc.memory.availableMB‚ 5.16796875 disk.sda3.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1108.6772499084473 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 158.40625 disk.nvme0n1.in‚ 158.40625 disk.dm-0.in‚ 158.40625 proc.cpu.threads‚17 network.recv‚649227' disk./.usagePercent‚74.404260483529 disk.sda.in‚0 disk.sda3.out‚ 63.48046875 disk.rbd2.out‚37.75244140625 disk.drbd1777.in‚0$ memory_percent‚99.96845722198486 disk.rbd3.in‚0 disk.sda1.out‚0 cpu‚25.25252016808072 disk.sda2.in‚0 disk.dm-1.out‚ 63.48046875 disk.rbd1.in‚0! proc.memory.rssMB‚ 2864.015625ŒªD’Þ‚(:‡ ¥¦æÔÏÝçÕ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.248' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äbŽß‚(:ˆ ¥¦æÔýàñØ disk.rbd0.in‚0' disk.nvme0c0n1.out‚5723.22607421875 disk.dm-0.in‚ 158.40625" disk.dm-0.out‚5723.22607421875 disk.sda.out‚ 64.05859375) disk./.usagePercent‚74.42110654730462 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0 disk.rbd2.out‚37.75244140625 disk.dm-1.out‚ 64.05859375) proc.memory.percent‚17.48056411743164 network.recv‚655395 disk.sda1.out‚0 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2864.015625% disk./.usageGB‚1108.9282684326172 disk.sda3.out‚ 64.05859375 disk.rbd3.out‚0 cpu‚25.250721437858104 network.sent‚ 3.860897e+06 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 158.40625 disk.nvme0n1.in‚ 158.40625% disk.nvme0n1.out‚5723.22607421875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 proc.memory.availableMB‚5.25$ memory_percent‚99.96795654296875ñÔý2à- _wandbruntime‚1674  _runtime‚1674m =ÇáÁ & lr-AdamW/pg2‚0.0001805458313403845 trainer/global_step‚2199 % _timestamp‚1.7884495828366752e+09 _runtime‚1674.391084159 _step‚175 & lr-AdamW/pg1‚0.0001805458313403845kt+ÌâÆ & lr-AdamW/pg1‚0.0001805458313403845 & lr-AdamW/pg2‚0.0001805458313403845 trainer/global_step‚2199 % _timestamp‚1.7884495828366752e+09 _runtime‚1674.391084159 _step‚175¯…£dQ2ã- _wandbruntime‚1674  _runtime‚1674ÉQ`MÙ'äÓO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139957495703552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 % _timestamp‚1.7884495828374815e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28800443 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 _step‚176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138038449382912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚1674.392206334 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41222708 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663~sŽÍÑåØO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28800443 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128140690641408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 _step‚176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31523576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138038449382912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚1674.392206334 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41222708 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124848376287663 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139957495703552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826800875520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 % _timestamp‚1.7884495828374815e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884495835409217e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10420405 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128197863121408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 _step‚177 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826963941888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140019996728320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128197188743680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827620316672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚1675.095951935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138100100120576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062ä—gè&èâM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826963941888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140019996728320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128197188743680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827620316672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚1675.095951935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138100100120576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚2199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31537590 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884495835409217e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10420405 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128197863121408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31537592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31537590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 _step‚177±,"'2é- _wandbruntime‚1675  _runtime‚1675V?ø&TêÎ 1loss/SRqqt3/detection_loss‚1.6875593763376986 trainer/global_step‚2199 _step‚178 0loss/FRt1/assignment_loss‚3.7496010369724697 1loss/SRqqt1/detection_loss‚2.0062082929582132 0loss/SRqqt2/detection_loss‚1.383201819844544 /loss/FBt1/detection_loss‚2.9703701734542847 $ _timestamp‚1.788449583542817e+09 /loss/FRt4/assignment_loss‚5.032545916736126 1loss/SRqqt2/assignment_loss‚1.748399590090331 2loss/SRqqt3/assignment_loss‚2.1888600446816002 0loss/FBt3/assignment_loss‚0.6325502837733941 -loss/FBt3/detection_loss‚2.27826861645046 2loss/SRqqt1/assignment_loss‚1.6060514604536498 .loss/FRt3/detection_loss‚2.210679673552513 %loss/total_loss‚51.03467168075946 /loss/FBt2/assignment_loss‚0.289414594677843 /loss/FBt2/detection_loss‚0.7961412352236018 /loss/FBt4/detection_loss‚3.2696086981079797 epoch‚1 _runtime‚1675.097149583 0loss/FBt1/assignment_loss‚0.6653637836046377 0loss/FBt4/assignment_loss‚0.4428209947390345 /loss/FRt2/detection_loss‚1.6821874730288982 .loss/FRt4/detection_loss‚2.942316334694624 1loss/SRqqt4/detection_loss‚2.7273814188582555 /loss/FRt3/assignment_loss‚3.753110749721527 0loss/FRt2/assignment_loss‚2.2041274468104044 2loss/SRqqt4/assignment_loss‚2.3054234805916036 .loss/FRt1/detection_loss‚2.462479185395771c~R1YëÓ trainer/global_step‚2199 _step‚178 0loss/FRt1/assignment_loss‚3.7496010369724697 1loss/SRqqt1/detection_loss‚2.0062082929582132 0loss/SRqqt2/detection_loss‚1.383201819844544 /loss/FBt1/detection_loss‚2.9703701734542847 $ _timestamp‚1.788449583542817e+09 /loss/FRt4/assignment_loss‚5.032545916736126 1loss/SRqqt2/assignment_loss‚1.748399590090331 2loss/SRqqt3/assignment_loss‚2.1888600446816002 0loss/FBt3/assignment_loss‚0.6325502837733941 -loss/FBt3/detection_loss‚2.27826861645046 2loss/SRqqt1/assignment_loss‚1.6060514604536498 .loss/FRt3/detection_loss‚2.210679673552513 %loss/total_loss‚51.03467168075946 /loss/FBt2/assignment_loss‚0.289414594677843 /loss/FBt2/detection_loss‚0.7961412352236018 /loss/FBt4/detection_loss‚3.2696086981079797 epoch‚1 _runtime‚1675.097149583 0loss/FBt1/assignment_loss‚0.6653637836046377 0loss/FBt4/assignment_loss‚0.4428209947390345 /loss/FRt2/detection_loss‚1.6821874730288982 .loss/FRt4/detection_loss‚2.942316334694624 1loss/SRqqt4/detection_loss‚2.7273814188582555 /loss/FRt3/assignment_loss‚3.753110749721527 0loss/FRt2/assignment_loss‚2.2041274468104044 2loss/SRqqt4/assignment_loss‚2.3054234805916036 .loss/FRt1/detection_loss‚2.462479185395771 1loss/SRqqt3/detection_loss‚1.6875593763376986²ž)xw“ì‚(:ˆ ´¦æÔϸÖ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝæÈEí‚(: ´¦æÔàóÚ$ memory_percent‚99.96793270111084 network.sent‚ 3.867305e+06 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚64.90625 disk.rbd2.out‚ 37.9013671875& disk.nvme0c0n1.out‚5820.8291015625 disk.sda2.out‚0 disk.dm-1.out‚64.90625 disk.rbd3.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚ 158.40625$ disk.nvme0n1.out‚5820.8291015625 disk.drbd1777.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.42103819329297 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚64.90625 disk.nvme0n1.in‚ 158.40625 disk.drbd1777.out‚0) proc.memory.percent‚17.48056411743164 network.recv‚660399 disk.dm-0.in‚ 158.40625 disk.dm-1.in‚0! proc.memory.rssMB‚ 2864.015625% disk./.usageGB‚1108.9272499084473! diÚaubÇsk.dm-0.out‚5820.8291015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 5.25390625 cpu‚25.249065504011018Š5Ï““î‚(:ˆ ææÔ›ÓÅÕ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.474' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O2ë^ï‚(:‹ ææÔ¸Š®Ø disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.recv‚666105! disk.nvme0c0n1.in‚ 158.4140625 disk.sda2.in‚0) proc.memory.percent‚17.48056411743164% disk./.usageGB‚1108.9246788024902 disk.rbd3.out‚0 disk.sda3.out‚ 65.84765625% disk.nvme0c0n1.out‚5823.185546875 disk.nvme0n1.in‚ 158.4140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 37.9130859375 disk.dm-0.out‚5823.185546875% proc.memory.availableMB‚ 5.3203125 cpu‚25.247238170655386 disk.rbd1.in‚0 disk.drbd1777.in‚0$ memory_percent‚99.96752738952637 network.sent‚ 3.975117e+06( disk./.usagePercent‚74.4208656442149 disk.dm-1.in‚0 disk.dm-1.out‚ 65.84765625 disk.sda.out‚ 65.84765625 disk.dm-0.in‚ 158.4140625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚5823.185546875! proc.memory.rssMB‚ 2864.015625zj3æ’ð‚(:‡ Ò¦æÔÒ´âÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ]óñ‚(: Ò¦æÔÎÔ¡Ø disk.sda2.out‚0 disk.sda.out‚ 66.421875 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 161.19921875 disk.nvme0n1.in‚ 161.19921875% disk.nvme0n1.out‚5935.22705078125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 39.6123046875" disk.dm-0.out‚5935.22705078125% disk./.usageGB‚1108.9358215332031 disk.rbd3.out‚0 disk.dm-1.out‚ 66.421875 cpu‚25.2455363631865) disk./.usagePercent‚74.42161344222242 disk.rbd1.out‚ 0.919921875 disk.sda1.in‚0% proc.memory.availableMB‚ 4.8359375 disk.sda2.in‚0 network.recv‚670445 disk.rbd3.in‚0 disk.sda3.out‚ 66.421875! proc.memory.rssMB‚ 2864.015625 disk.sda.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 161.19921875 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚99.97048377990723) proc.memory.percent‚17.48056411743164 network.sent‚ 3.981233e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚5935.22705078125¼ÄN2ò- _wandbruntime‚1712  _runtime‚1712]L|¯Éóà ' lr-AdamW/pg2‚0.00018016157699607773 trainer/global_step‚2249 % _timestamp‚1.7884496207428892e+09 _runtime‚1712.297390115 _step‚179 ' lr-AdamW/pg1‚0.00018016157699607773z.nÎôÈ ' lr-AdamW/pg1‚0.00018016157699607773 ' lr-AdamW/pg2‚0.00018016157699607773 trainer/global_step‚2249 % _timestamp‚1.7884496207428892e+09 _runtime‚1712.297390115 _step‚179³G$‡2õ- _wandbruntime‚1712  _runtime‚1712s,ÐÙ'öÓO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32224376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130999328389632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127634418155071 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1962038626304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867130319872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884496207437074e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10646602 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867786681344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 _runtime‚1712.298304358 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42128724 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0VÁÙÞ'÷ØO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10646602 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867786681344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129131541708288 _runtime‚1712.298304358 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42128724 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32224376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125773501035520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157333504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130999328389632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127634418155071 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129131541708288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1962038626304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130999328389632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867130319872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884496207437074e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860262681803 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944´úë2ø- _wandbruntime‚1713  _runtime‚1713”éij§!ùÜM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32238392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42167240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42167240 _step‚181 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125828405166080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788449621466961e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1868606122496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2249 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 ^JDeviceStatsMoni|œN;tor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1867949747712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728‘žð}ç&úáM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32238392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131056500869632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42167240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42167240 _step‚181 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125828405166080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788449621466961e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1868606122496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2249 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1867949747712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424µa‘¡2û- _wandbruntime‚1713  _runtime‚1713­§¥ËYüÓ 2loss/SRqqt3/assignment_loss‚1.7629640060801839 .loss/FRt3/detection_loss‚2.366289751755225 /loss/FRt1/detection_loss‚2.4359439139096244 /loss/FBt4/detection_loss‚3.4225907921791077 %loss/total_loss‚54.48512377847646 epoch‚1 _step‚182 0loss/SRqqt1/detection_loss‚2.176471132446419 /loss/FBt3/detection_loss‚2.7353047827879586 _runtime‚1713.023203509 0loss/FRt3/assignment_loss‚3.8341277177269393 1loss/FBt3/assignment_loss‚0.49141094460997675 /loss/FRt2/detection_loss‚2.0772382400783838 1loss/FBt4/assignment_loss‚0.49720073491334915 0loss/SRqqt3/detection_loss‚1.984391411376554 1loss/SRqqt4/detection_loss‚2.4207060874962227 .loss/FBt1/detection_loss‚5.510603308677673 % _timestamp‚1.7884496214688692e+09 /loss/FRt2/assignment_loss‚1.998302134813047 /loss/FRt4/assignment_loss‚5.238555073738098 2loss/SRqqt1/assignment_loss‚1.7015063234350898 2loss/SRqqt4/assignment_loss‚2.5931267244059866 0loss/FBt1/assignment_loss‚0.7297148257493973 .loss/FRt4/detection_loss‚3.073104958449091 /loss/FBt2/detection_loss‚0.7979537072605167 0loss/FBt2/assignment_loss‚0.2051701137609536 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚4.072493278755332 2loss/SRqqt2/assignment_loss‚1.2024264189682594 1loss/SRqqt2/detection_loss‚1.1575273951030758•ÒK^ýØ 0loss/FBt2/assignment_loss‚0.2051701137609536 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚4.072493278755332 2loss/SRqqt2/assignment_loss‚1.2024264189682594 1loss/SRqqt2/detection_loss‚1.1575273951030758 2loss/SRqqt3/assignment_loss‚1.7629640060801839 .loss/FRt3/detection_loss‚2.366289751755225 /loss/FRt1/detection_loss‚2.4359439139096244 /loss/FBt4/detection_loss‚3.4225907921791077 %loss/total_loss‚54.48512377847646 epoch‚1 _step‚182 0loss/SRqqt1/detection_loss‚2.176471132446419 /loss/FBt3/detection_loss‚2.7353047827879586 _runtime‚1713.023203509 0loss/FRt3/assignment_loss‚3.8341277177269393 1loss/FBt3/assignment_loss‚0.49141094460997675 /loss/FRt2/detection_loss‚2.0772382400783838 1loss/FBt4/assignment_loss‚0.49720073491334915 0loss/SRqqt3/detection_loss‚1.984391411376554 1loss/SRqqt4/detection_loss‚2.4207060874962227 .loss/FBt1/detection_loss‚5.510603308677673 % _timestamp‚1.7884496214688692e+09 /loss/FRt2/assignment_loss‚1.998302134813047 /loss/FRt4/assignment_loss‚5.238555073738098 2loss/SRqqt1/assignment_loss‚1.7015063234350898 2loss/SRqqt4/assignment_loss‚2.5931267244059866 0loss/FBt1/assignment_loss‚0.7297148257493973 .loss/FRt4/detection_loss‚3.073104958449091 /loss/FBt2/detection_loss‚0.7979537072605167¶|®‰’þ‚(:‡ á¦æÔí¯ÕÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j*=ÿ‚(:“ á¦æÔ±À‹Ù disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 67.12890625) proc.memory.percent‚17.48056411743164 cpu‚25.24397939817651 disk.dm-0.in‚ 161.19921875" disk.nvme0c0n1.in‚ 161.19921875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 39.6279296875$ proc.memory.availableMB‚5.265625 disk.dm-1.out‚ 67.12890625 disk.rbd1.out‚ 0.919921875$ memory_percent‚99.96786117553711! proc.memory.rssMB‚ 2864.015625" disk.dm-0.out‚6032.90869140625 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚6032.90869140625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚6032.90869140625 proc.cpu.threads‚17 network.recv‚674768% disk./.usageGB‚1108.9358253479004 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 161.19921875 network.sent‚ 3.986776e+06) disk./.usagePercent‚74.42161369822995 disk.sda.out‚ 67.12890625 disk.rbd2.in‚0Nû€‚(:’ ð¦æÔ“»µÕ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.287999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó äd‚(:ý ð¦æÔ„³ÑØ network.sent‚ 4.09538e+06 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 6034.5703125 disk.sda.out‚ 67.6015625 disk.dm-0.in‚ 162.21875% disk./.usageGB‚1108.9364700317383 disk.rbd1.out‚ 0.919921875# disk.nvme0c0n1.out‚ 6034.5703125 disk.sda2.in‚0! proc.memory.rssMB‚ 2864.015625 cpu‚25.242486731311846) disk./.usagePercent‚74.42165696350324 disk.dm-1.out‚ 67.6015625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.out‚ 6034.5703125 network.recv‚681273 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 5.14453125) proc.memory.percent‚17.48056411743164 disk.nvme0n1.in‚ 162.21875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 67.6015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 39.7529296875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 162.21875$ memory_percent‚99.96860027313232*Ëp^2‚- _wandbruntime‚1749  _runtime‚1749ºDš9ǃÁ & lr-AdamW/pg1‚0.0001797589795426256 & lr-AdamW/pg2‚0.0001797589795426256 trainer/global_step‚2299 % _timestamp‚1.7884496575999894e+09 _runtime‚1749.154433096 _step‚183EóOÌ„Æ _step‚183 & lr-AdamW/pg1‚0.0001797589795426256 & lr-AdamW/pg2‚0.0001797589795426256 trainer/global_step‚2299 % _timestamp‚1.7884496575999894e+09 _runtime‚1749.154433096·Ñ D2…- _wandbruntime‚1749  _runtime‚1749´•äùÙ'†ÓO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2005010566656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130419788545371 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32925176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131949175647744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146208533715968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146208341956608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144203523149312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128518690524160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1908772487168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884496576009548e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144203499209216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10874947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2004842747392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1901098021211 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361äþDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144203499209216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10874947 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2004842747392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1901098021211 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚kº Öw0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32925176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2005010566656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130419788545371 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¸SþV:2ˆ- _wandbruntime‚1749  _runtime‚1749KAâ&‰ÜM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2005865917440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32939192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10879025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146270850155520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1908935553536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133914464240128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76032484 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1902568894131 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128573611694080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884496582873516e+09 _step‚185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132005546689536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1908935553536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43073230 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144265149946880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230ýtç&ŠáM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1909591928320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132005546689536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1908935553536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43073230 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144265149946880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2005865917440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32939192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10879025 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146270850155520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1908935553536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133914464240128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76032484 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1902568894131 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128573611694080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884496582873516e+09 _step‚185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928¹'‰Ïp2‹- _wandbruntime‚1749  _runtime‚1749.ˆ”XŒÒ 2loss/SRqqt1/assignment_loss‚1.4918409382424704 /loss/FBt1/assignment_loss‚0.989527195536842 .loss/FRt2/detection_loss‚2.064630212431604 1loss/SRqqt4/detection_loss‚2.8335702562332155 .loss/FBt4/detection_loss‚3.427147856780461 epoch‚1 .loss/FRt1/detection_loss‚2.652873066564401 0loss/FRt3/assignment_loss‚3.9605162751440908 /loss/FRt4/assignment_loss‚5.837836452892849 2loss/SRqqt3/assignment_loss‚1.6365773623639888 0loss/FBt3/assignment_loss‚0.5791117423637346 _runtime‚1749.843720641 _step‚186 0loss/FRt1/assignment_loss‚3.3682388183143406 /loss/FRt3/detection_loss‚2.1904084676621007 /loss/FBt1/detection_loss‚2.0562138160069785 % _timestamp‚1.7884496582894254e+09 0loss/FRt2/assignment_loss‚1.4753170230171897 2loss/SRqqt4/assignment_loss‚2.5129519790410995 1loss/SRqqt2/detection_loss‚1.3389600169274114 .loss/FBt3/detection_loss‚2.196887493133545 trainer/global_step‚2299 0loss/FBt2/assignment_loss‚0.2379983544546697 1loss/SRqqt3/detection_loss‚1.9000929550691084 0loss/SRqqt1/detection_loss‚2.077051430213742 /loss/FBt2/detection_loss‚0.8347726404299394 2loss/SRqqt2/assignment_loss‚1.3110699613969172 0loss/FBt4/assignment_loss‚0.6230943713869286 /loss/FRt4/detection_loss‚3.0552049704960416 %loss/total_loss‚50.65189365610367ÞÒ š]× 0loss/FBt4/assignment_loss‚0.6230943713869286 /loss/FRt4/detection_loss‚3.0552049704960416 %loss/total_loss‚50.65189365610367 2loss/SRqqt1/assignment_loss‚1.4918409382424704 /loss/FBt1/assignment_loss‚0.989527195536842 .loss/FRt2/detection_loss‚2.064630212431604 1loss/SRqqt4/detection_loss‚2.8335702562332155 .loss/FBt4/detection_loss‚3.427147856780461 epoch‚1 .loss/FRt1/detection_loss‚2.652873066564401 0loss/FRt3/assignment_loss‚3.9605162751440908 /loss/FRt4/assignment_loss‚5.837836452892849 2loss/SRqqt3/assignment_loss‚1.6365773623639888 0loss/FBt3/assignment_loss‚0.5791117423637346 _runtime‚1749.843720641 _step‚186 0loss/FRt1/assignment_loss‚3.3682388183143406 /loss/FRt3/detection_loss‚2.1904084676621007 /loss/FBt1/detection_loss‚2.0562138160069785 % _timestamp‚1.7884496582894254e+09 0loss/FRt2/assignment_loss‚1.4753170230171897 2loss/SRqqt4/assignment_loss‚2.5129519790410995 1loss/SRqqt2/detection_loss‚1.3389600169274114 .loss/FBt3/detection_loss‚2.196887493133545 trainer/global_step‚2299 0loss/FBt2/assignment_loss‚0.2379983544546697 1loss/SRqqt3/detection_loss‚1.9000929550691084 0loss/SRqqt1/detection_loss‚2.077051430213742 /loss/FBt2/detection_loss‚0.8347726404299394 2loss/SRqqt2/assignment_loss‚1.3110699613969172ºp€º“Ž‚(:ˆ ÿ¦æÔ¿©ÉÕ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚJí4‚(:ý ÿ¦æÔ»ŒÌØ disk.sda.out‚ 68.3203125 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 39.7685546875$ disk./.usageGB‚1109.436279296875 disk.sda3.out‚ 68.3203125 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚684701) disk./.usagePercent‚74.45519958265447 disk.dm-0.in‚ 162.21875 disk.sda2.in‚0 disk.dm-0.out‚ 6132.0859375 disk.dm-1.in‚0 disk.dm-1.out‚ 68.3203125$ memory_percent‚99.94776248931885& proc.memory.availableMB‚ 8.55859375 proc.cpu.threads‚17! disk.nvme0n1.out‚ 6132.0859375 network.sent‚ 4.100657e+06 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.21875 cpu‚25.240866085265637 disk.rbd3.in‚0) proc.memory.percent‚17.48056411743164 disk.sda.in‚0 disk.nvme0c0n1.in‚ 162.21875# disk.nvme0c0n1.out‚ 6132.0859375! proc.memory.rssMB‚ 2864.015625ƒ@“‚(:ˆ Ž§æÔìÑÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑLz‘‚(:† Ž§æÔÞû€Ù disk.sda2.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1777.in‚0! disk.dm-0.out‚6132.7978515625! proc.memory.rssMB‚ 2864.015625) proc.memory.percent‚17.48056411743164) disk./.usagePercent‚74.45521007896338$ disk./.usageGB‚1109.436435699463 disk.sda.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚162.9375& disk.nvme0c0n1.out‚6132.7978515625 disk.dm-0.in‚162.9375$ memory_percent‚99.94854927062988 network.sent‚ 4.209079e+06 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 68.67578125% proc.memory.availableMB‚ 8.4296875 proc.cpu.threads‚17 network.recv‚691201 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 68.67578125 disk.rbd2.out‚39.85498046875 disk.nvme0n1.in‚162.9375$ disk.nvme0n1.out‚6132.7978515625 disk.drbd1777.out‚0 cpu‚25.239259719316472 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 68.67578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0ûÊLž’‚(:“ §æÔîŽÖ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.973200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(^»Æ “‚(:þ §æÔóã£Ø disk.nvme0c0n1.in‚162.9375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚74.45521391907639 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 69.328125 disk.rbd0.out‚0 disk.dm-0.in‚162.9375& disk.nvme0c0n1.out‚6230.2431640625$ disk.nvme0n1.out‚6230.2431640625 disk.rbd3.in‚0! disk.dm-0.out‚6230.2431640625 disk.dm-1.out‚ 69.328125 disk.sda1.out‚0# memory_percent‚99.9485969543457 disk.nvme0n1.in‚162.9375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 network.sent‚ 4.214548e+06 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 69.328125 disk.rbd2.out‚41.44482421875) proc.memory.percent‚17.48056411743164 disk.sda1.in‚0 network.recv‚695358% disk./.usageGB‚1109.4364929199219 disk.drbd1777.in‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚8.421875! proc.memory.rssMB‚ 2864.015625 cpu‚25.23771278945155ÎÖc$2”- _wandbruntime‚1786  _runtime‚1786ñ¡àsÉ•Ã ' lr-AdamW/pg1‚0.00017933812288073404 ' lr-AdamW/pg2‚0.00017933812288073404 trainer/global_step‚2349 % _timestamp‚1.7884496952709646e+09 _runtime‚1786.825408484 _step‚187½~Y„V–È ' lr-AdamW/pg1‚0.00017933812288073404 ' lr-AdamW/pg2‚0.000179338122880734Ćx04 trainer/global_step‚2349 % _timestamp‚1.7884496952709646e+09 _runtime‚1786.825408484 _step‚187»º¡ún2—- _wandbruntime‚1786  _runtime‚1786˜w1Ù'˜ÓO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133205830412779 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33625976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44000864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136716603886080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 % _timestamp‚1.7884496952717144e+09 _runtime‚1786.826430259 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚188 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136716603886080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1941933360619 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134766845593088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30714427 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147286060032512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43980828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134766827590144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134766827590144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1949101931520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149333833541120 trainer/global_step‚2349 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11104175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1949758292992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11104624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 €¸ÆÞ'™ØO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77606802 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2047797448704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133206501889887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33625976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19610252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1949758292992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11104624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1942587798367 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133205830412779 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33625976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44000864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136716603886080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 % _timestamp‚1.7884496952717144e+09 _runtime‚1786.826430259 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚188 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136716603886080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1941933360619 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134766845593088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30714427 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19610254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147286060032512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43980828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134766827590144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134766827590144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1949101931520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149333833541120 trainer/global_step‚2349 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11104175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149396344464896 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134823180628992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147347710770176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2048821246464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2048657634816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1950577734144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚189 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11109152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1950577734144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19618422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884496959748802e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33639990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149396344464896 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134823180628992 C9DeviceStatsMonitor.on_tR–…rain_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147347710770176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2048821246464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2048657634816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1950577734144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚189 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11109152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1950577734144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19618422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884496959748802e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137584479532544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11334632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31352831 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150368596915712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139574567269888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137584497535488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137584479532544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983423137775 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34326776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152459529724416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135991872280187 % _timestamp‚1.7884497328169096e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1824.371569729 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 Á,!©ØO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135991872280187 % _timestamp‚1.7884497328169096e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1824.371569729 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1990087737344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139575241634304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137584479532544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1990744098816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11334632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31352831 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150368596915712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139574567269888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137584497535488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.free‹×2½Ãd‚137584479532544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983423137775 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34326776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152459529724416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1982768700027 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928Àr”¸¸2ª- _wandbruntime‚1825  _runtime‚1825ÞQˆ·ã&«ÝM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 _step‚193 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991563539968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1983585134991 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152522229285376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚1825.066745596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2091818020352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124(hM[è&¬âM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136048264323027 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136047592845711 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152522229285376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚1825.066745596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2091818020352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20026820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31366077 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 _step‚193 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991563539968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1983585134991 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚153395847932928 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2016770719863 % _timestamp‚1.7884497955675678e+09 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚45619235 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35122574 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35122572[{jšE*º¿T LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚45619235 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35122574 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35122572 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚153395871873024 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚150 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113247744 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953744 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚138797956081191 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136781676275712 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832128 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45635210 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2023728806912 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510832128 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚2417 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚142483587488768 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80741807 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _runtime‚1887.122434665 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚153395847932928 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2016770719863 % _timestamp‚1.7884497955675678e+09Ãýf\2»- _wandbruntime‚1888  _runtime‚1888OÒÁI)¼ÃR ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2016643414887 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832128 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 _step‚196 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚150 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚140496706713600 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _runtime‚1888.040826455 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585024 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚140496688710656 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 $ _timestamp‚1.788449796486005e+09 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚142521293476352 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414¥)º ½ÈR OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚142521293476352 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚45642512 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2016643414887 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24399 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002.¥)E944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚11646720 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832128 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 _step‚196 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2024093933568 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚150 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚140496706713600 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚142520782644224 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _runtime‚1888.040826455 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585024 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2024586762752 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚140496688710656 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚155559701253120 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 $ _timestamp‚1.788449796486005e+09 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915Äö¥ã(‘¾‚(:† †¨æÔ¼áÕ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Bp-¿‚(:‰ †¨æÔÃÕØ cpu‚25.359875175172125 network.sent‚ 4.458885e+06 network.recv‚728483 disk.sda2.out‚0 disk.sda.out‚ 73.73046875 disk.rbd1.out‚ 0.919921875 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6525.50341796875% disk.nvme0n1.out‚6525.50341796875 disk.drbd1777.out‚0$ memory_percent‚99.97637271881104) disk./.usagePercent‚74.45521212702366 disk.sda3.out‚ 73.73046875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚162.9375* proc.memory.percent‚17.511963844299316 proc.cpu.threads‚17 disk.dm-1.out‚ 73.73046875 disk.rbd0.in‚0& proc.memory.availableMB‚ 3.87109375 disk.rbd1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1109.436466217041 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2869.16015625 disk.dm-0.in‚162.9375" disk.dm-0.out‚6525.50341796875 disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 43.3984375 disk.nvme0n1.in‚162.9375 disk.drbd1777.in‚0-¢2À- _wandbruntime‚1900  _runtime‚1900ä ñGÁ[Áͦ ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4500718116760254 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15551427006721497 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37816092371940613 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11532130092382431 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3190789520740509 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5018735527992249 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.46712562441825867 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5450533628463745 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004535147454589605 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1682615429162979 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04764018580317497 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 *jet/accuracy_2_of_2‚0.0747244656085968 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9148936867713928 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9887697100639343 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11995608359575272 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41070109605789185 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3205887973308563 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9540229439735413 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9817963242530823 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.977846086025238 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19162723422050476 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28704342246055603 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05798812955617905 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994175374507904 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9857684373855591 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4533243477344513 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.10787560790777206 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3855750560760498 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1935136765241623 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19564029574394226 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/f_score‚0.30171123147010803 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884498091932127e+09 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32667553424835205 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4652043581008911 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.45679011940956116 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44255614280700684 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12040512263774872 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 particle/accuracy_6_of_7‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3274737000465393 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01075268816202879 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11500024795532227 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9845613837242126 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41537362337112427 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3893246054649353 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4233909249305725 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.974983811378479 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.311686635017395 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9860044717788696 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1783040165901184 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1364864856004715 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.456079363822937 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1609259694814682 +jet/accuracy_1_of_1‚0.44888243079185486 *jet/accuracy_1_of_4‚0.8737339973449707 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36268872022628784 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10650104284286499 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44756945967674255 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4518526792526245 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2767241299152374 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 (particle/accuracy‚0.8304399847984314 +particle/sensitivity‚0.2347186952829361 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870370745658875 jet/accuracy_5_of_6‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4584433138370514 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21666699647903442 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9976985454559326 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07551202178001404 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4722367525100708 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4123409688472748 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27704715728759766 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2529761791229248 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36639338731765747 particle/accuracy_2_of_3‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007251082453876734 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_1_of_7‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35045161843299866 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05774910748004913 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.43994420766830444 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10972120612859726 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9406564831733704 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3958333730697632 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39226576685905457 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4719061255455017 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9194508194923401 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777934551239014 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.049637727439403534 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.036610934883356094 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.007236369885504246 particle/accuracy_2_of_2‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09003213047981262 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006122448947280645 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 particle/accuracy_4_of_5‚1 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.307407408952713 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03742959722876549 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4280704855918884 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 _step‚197 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43264636397361755 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41606754064559937 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3828575611114502 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5373912453651428 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 jet/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.47413793206214905 particle/accuracy_1_of_1‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5077381134033203 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02466628886759281 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33803024888038635 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4895886778831482 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05093245953321457 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34687498211860657 +jet/accuracy_2_of_4‚0.29338422417640686 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872908592224121 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9951889514923096 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05648508667945862 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.006853735540062189 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014041514368727803 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31589147448539734 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1308564692735672 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.408698707818985 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10661547631025314 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.012944276444613934 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_5_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10661547631025314 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_5_of_5‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31344375014305115 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9581604599952698 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02132449485361576 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3346979022026062 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190462589263916 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 5 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17496258020401 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9864566326141357 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45719197392463684 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07974138110876083 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.44691988825798035 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11430684477090836 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14040403068065643 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.38698381185531616 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36532068252563477 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17438273131847382 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911340475082397 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07444053143262863 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11315034329891205 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2386859655380249 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4561886191368103 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14300410449504852 particle/accuracy_1_of_4‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0025428244844079018 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912943840026855 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14076974987983704 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_4_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0Ä·7¡“7.9583333134651184 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007071382366120815 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5254237055778503 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3462797701358795 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3771354556083679 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3308638036251068 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1675562709569931 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9799139499664307 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4275793433189392 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11696524173021317 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.7106900811195374 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.2884615361690521 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998525857925415 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43083587288856506 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9674608111381531 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17096544802188873 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31871354579925537 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49425235390663147 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.40757521986961365 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4366956651210785 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_5_of_5‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46632203459739685 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3055059313774109 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_6‚0.9230769276618958 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4051673412322998 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44166436791419983 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.42708340287208557 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9838042855262756 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12458821386098862 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005272621754556894 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1402312070131302 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.45443159341812134 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34144142270088196 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚2417 _runtime‚1900.750391482 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923380017280579 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49679484963417053 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13930954039096832 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.46898409724235535 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12160492688417435 7validation_average_jet_accuracy‚0.39909398555755615 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚51.59182355083161 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0660175234079361 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25172343850135803 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.37418070435523987 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6292163133621216 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13965517282485962 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 +jet/accuracy_2_of_3‚0.19516783952713013 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004535147454589605 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4795759916305542 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43335169553756714 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9625991582870483 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16932083666324615 epoch‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11261054128408432 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4350953996181488 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4360036551952362 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9810091257095337 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4456402063369751 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *jet/accuracy_1_of_3‚0.8200239539146423 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11503471434116364 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08554420620203018 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9745711088180542 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1931876242160797 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45719197392463684 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3712121248245239 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691343307495 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 jet/accuracy_5_of_7‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11871345341205597 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35841745138168335 particle/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4680136442184448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5349207520484924 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.111111119389534 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6225736141204834 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9976985454559326 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1969166100025177 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9948107600212097 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45180386304855347 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.13461539149284363 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13179495930671692 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 jet/accuracy_4_of_6‚01&§å_HÂÒ¦ 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 *jet/accuracy_2_of_2‚0.0747244656085968 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9148936867713928 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9887697100639343 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11995608359575272 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41070109605789185 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3205887973308563 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9540229439735413 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9817963242530823 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.977846086025238 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19162723422050476 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28704342246055603 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05798812955617905 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994175374507904 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9857684373855591 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4533243477344513 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.10787560790777206 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3855750560760498 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1935136765241623 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19564029574394226 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/f_score‚0.30171123147010803 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884498091932127e+09 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32667553424835205 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4652043581008911 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.45679011940956116 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44255614280700684 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12040512263774872 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 particle/accuracy_6_of_7‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3274737000465393 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01075268816202879 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11500024795532227 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9845613837242126 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41537362337112427 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3893246054649353 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4233909249305725 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.974983811378479 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.311686635017395 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9860044717788696 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1783040165901184 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1364864856004715 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.456079363822937 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1609259694814682 +jet/accuracy_1_of_1‚0.44888243079185486 *jet/accuracy_1_of_4‚0.8737339973449707 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36268872022628784 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10650104284286499 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44756945967674255 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4518526792526245 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2767241299152374 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 (particle/accuracy‚0.8304399847984314 +particle/sensitivity‚0.2347186952829361 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870370745658875 jet/accuracy_5_of_6‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4584433138370514 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21666699647903442 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.190476194024086 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9976985454559326 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07551202178001404 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4722367525100708 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4123409688472748 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27704715728759766 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2529761791229248 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36639338731765747 particle/accuracy_2_of_3‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007251082453876734 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_1_of_7‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35045161843299866 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05774910748004913 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.43994420766830444 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10972120612859726 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9406564831733704 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3958333730697632 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39226576685905457 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4719061255455017 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9194508194923401 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777934551239014 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.049637727439403534 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.036610934883356094 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.007236369885504246 particle/accuracy_2_of_2‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09003213047981262 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006122448947280645 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 particle/accuracy_4_of_5‚1 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.307407408952713 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03742959722876549 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4280704855918884 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 _step‚197 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43264636397361755 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41606754064559937 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3828575611114502 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5373912453651428 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 jet/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.47413793206214905 particle/accuracy_1_of_1‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5077381134033203 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02466628886759281 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33803024888038635 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4895886778831482 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05093245953321457 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34687498211860657 +jet/accuracy_2_of_4‚0.29338422417640686 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872908592224121 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9951889514923096 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.05648508667945862 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.006853735540062189 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014041514368727803 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31589147448539734 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1308564692735672 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.408698707818985 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10661547631025314 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.012944276444613934 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_5_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10661547631025314 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 jet/accuracy_5_of_5‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31344375014305115 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9581604599952698 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02132449485361576 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457ä÷G°úJ397 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3346979022026062 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190462589263916 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 5 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17496258020401 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9864566326141357 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45719197392463684 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07974138110876083 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.44691988825798035 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11430684477090836 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14040403068065643 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.38698381185531616 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36532068252563477 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17438273131847382 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911340475082397 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07444053143262863 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11315034329891205 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2386859655380249 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4561886191368103 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14300410449504852 particle/accuracy_1_of_4‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0025428244844079018 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9912943840026855 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14076974987983704 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_4_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007071382366120815 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5254237055778503 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3462797701358795 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3771354556083679 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3308638036251068 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1675562709569931 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9799139499664307 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4275793433189392 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11696524173021317 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.7106900811195374 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.2884615361690521 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998525857925415 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43083587288856506 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9674608111381531 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17096544802188873 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31871354579925537 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49425235390663147 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.40757521986961365 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4366956651210785 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_5_of_5‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46632203459739685 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3055059313774109 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_6‚0.9230769276618958 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4051673412322998 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44166436791419983 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.42708340287208557 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9838042855262756 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12458821386098862 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005272621754556894 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1402312070131302 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.45443159341812134 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34144142270088196 particle/accuracy_6_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚2417 _runtime‚1900.750391482 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923380017280579 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49679484963417053 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_3_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13930954039096832 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.46898409724235535 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12160492688417435 7validation_average_jet_accuracy‚0.39909398555755615 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚51.59182355083161 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0660175234079361 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25172343850135803 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.37418070435523987 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6292163133621216 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13965517282485962 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 +jet/accuracy_2_of_3‚0.19516783952713013 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666663646698 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004535147454589605 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4795759916305542 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43335169553756714 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9625991582870483 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16932083666324615 epoch‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11261054128408432 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4350953996181488 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4360036551952362 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9810091257095337 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4456402063369751 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *jet/accuracy_1_of_3‚0.8200239539146423 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01684636063873768 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11503471434116364 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08554420620203018 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9745711088180542 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1931876242160797 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45719197392463684 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3712121248245239 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691343307495 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 jet/accuracy_5_of_7‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11871345341205597 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35841745138168335 particle/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4680136442184448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5349207520484924 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.111111119389534 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6225736141204834 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9976985454559326 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1969166100025177 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9948107600212097 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45180386304855347 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.13461539149284363 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13179495930671692 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 jet/accuracy_4_of_6‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4500718116760254 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15551427006721497 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37816092371940613 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11532130092382431 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3190789520740509 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5018735527992249 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.46712562441825867 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5450533628463745 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004535147454589605 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1682615429162979 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04764018580317497 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1Å%œÂ(:‘ •¨æÔ²†Ö gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚75.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.186399999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0› Ä‚(:† •¨æÔ¬‚Ù) disk./.usagePercent‚74.50329955822521 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0$ disk./.usageGB‚1110.153003692627 disk.sda.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 3348.22265625 proc.cpu.threads‚17 disk.nvme0n1.in‚162.9375 disk.dm-1.out‚ 74.7109375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 74.7109375 disk.rbd0.in‚0 disk.rbd2.out‚ 43.4140625 disk.dm-0.in‚162.9375( proc.memory.availableMB‚ 307.22265625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚162.9375 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚6623.04833984375% disk.nvme0n1.out‚6623.04833984375 disk.drbd1777.in‚0 cpu‚25.38788825607461 network.recv‚734782" disk.dm-0.out‚6623.04833984375 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚98.12486171722412) proc.memory.percent‚20.43592929840088 disk.sda.out‚ 74.7109375 network.sent‚ 4.566435e+06VXÔx’Å‚(:‡ ¤¨æÔÅÖÃÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚95.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙѼ5Æ‚(:… ¤¨æÔíÕ‰Ù disk.dm-1.out‚ 75.4609375 network.sent‚ 4.690836e+06 disk.rbd3.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚7096.513671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.383996941005176$ memory_percent‚94.17941570281982 disk.rbd1.in‚0) proc.memory.percent‚17.54920482635498 disk.dm-0.in‚162.9375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 75.4609375 disk.nvme0n1.in‚162.9375# proc.memory.rssMB‚ 2875.26171875 disk.rbd0.in‚0# disk.nvme0n1.out‚7096.513671875 disk.drbd1777.in‚0 disk.dm-0.out‚7096.513671875( proc.memory.availableMB‚ 953.64453125 proc.cpu.threads‚17% disk./.usageGB‚1109.9001770019531 disk.rbd2.in‚0 disk.nvme0c0n1.in‚162.9375 network.recv‚741669) disk./.usagePercent‚74.48633214687837 disk.rbd1.out‚ 0.919921875 disk.rbd3.in‚0 disk.sda.out‚ 75.4609375 disk.rbd2.out‚ 43.5087890625KÚüï2Ç- _wandbruntime‚1929  _runtime‚1929ìȃÃÇÈÁ & lr-AdamW/pg1‚0.0001784419865421366 & lr-AdamW/pg2‚0.0001784419865421366 trainer/global_step‚2449 % _timestamp‚1.7884498375185766e+09 _runtime‚1929.072988664 _step‚198Oò± ÌÉÆ % _timestamp‚1.7884498375185766e+09 _runtime‚1929.072988664 _step‚198 & lr-AdamW/pg1‚0.0001784419865421366 & lr-AdamW/pg2‚0.0001784419865421366 trainer/global_step‚2449ÆÉ.°‘2Ê- _wandbruntime‚1929  _runtime‚1929ü€z'ËÓO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1929.074063239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2047182977261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142722423289344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46294890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20836806 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32682006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2054749872640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46314927 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚2449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141025016972525 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157994536023552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155837305009664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2054749872640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2157254953984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35708354 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138977816955904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884498375193706e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35708352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytªDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138977816955904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884498375193706e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35708352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1929.074063239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2047182977261 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142722423289344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46294890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20836806 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32682006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2054749872640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46314927 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚2449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141025016972525 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157994729879552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157994536023552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155837305009664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2054749872640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2157254953984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944Ç2ïÚ‹2Í- _wandbruntime‚1929  _runtime‚1929ä+»kã&ÎÝM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155898931807232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142778776328192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚1929.781250282 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141080737538257 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142778794331136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139032738125824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32694642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2158107078144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2158276980736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20844972 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712õŽ7è&ÏâM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141080737538257 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142778794331136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11849668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139032738125824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32694642 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2158107078144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2158276980736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158057232728064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46313344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20844972 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047999412433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155898931807232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142778776328192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2054912938496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚1929.781250282 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536ÈÍÇ)m2Ð- _wandbruntime‚1929  _runtime‚1929ˆ1Ë$ZÑÔ 1loss/SRqqt2/assignment_loss‚1.328995459542518 1loss/SRqqt2/detection_loss‚1.1654460503601691 .loss/FBt1/detection_loss‚3.103851463113512 /loss/FBt2/detection_loss‚0.7412714692394505 0loss/FRt4/assignment_loss‚4.9987792860377915 1loss/SRqqt3/detection_loss‚1.9937071064297034 1loss/SRqqt4/detection_loss‚2.5521904810478815 .loss/FBt4/detection_loss‚3.078168574968974 % _timestamp‚1.7884498382280138e+09 _step‚201 0loss/FRt2/assignment_loss‚1.4422097585418008 epoch‚2 trainer/global_step‚2449 0loss/FBt1/assignment_loss‚0.5554362152303968 1loss/FBt4/assignment_loss‚0.48971420851691266 .loss/FBt3/detection_loss‚2.626153381431804 2loss/SRqqt3/assignment_loss‚1.3773596880159207 1loss/FBt3/assignment_loss‚0.48665176136139743 .loss/FRt2/detection_loss‚2.006273979024792 %loss/total_loss‚48.31165001779124 /loss/FRt1/assignment_loss‚3.114036449030334 2loss/SRqqt1/assignment_loss‚1.4456868269953176 1loss/FBt2/assignment_loss‚0.28415613011023255 .loss/FRt1/detection_loss‚2.252363241186329 1loss/SRqqt1/detection_loss‚1.7084639163120934 0loss/FRt3/assignment_loss‚3.6965176423390704 2loss/SRqqt4/assignment_loss‚2.3613569156119696 .loss/FRt3/detection_loss‚2.238067852788501 /loss/FRt4/detection_loss‚3.2647921605543657 _runtime‚1929.782340547,™§©ÒÙ 0loss/FRt3/assignment_loss‚3.6965176423390704 2loss/SRqqt4/assignment_loss‚2.3613569156119696 .loss/FRt3/detection_loss‚2.238067852788501 /loss/FRt4/detection_loss‚3.2647921605543657 _runtime‚1929.782340547 1loss/SRqqt2/assignment_loss‚1.328995459542518 1loss/SRqqt2/detection_loss‚1.1654460503601691 .loss/FBt1/detection_loss‚3.103851463113512 /loss/FBt2/detection_loss‚0.7412714692394505 0loss/FRt4/assignment_loss‚4.9987792860377915 1loss/SRqqt3/detection_loss‚1.9937071064297034 1loss/SRqqt4/detection_loss‚2.5521904810478815 .loss/FBt4/detection_loss‚3.078168574968974 % _timestamp‚1.7884498382280138e+09 _step‚201 0loss/FRt2/assignment_loss‚1.4422097585418008 epoch‚2 trainer/global_step‚2449 0loss/FBt1/assignment_loss‚0.5554362152303968 1loss/FBt4/assignment_loss‚0.48971420851691266 .loss/FBt3/detection_loss‚2.626153381431804 2loss/SRqqt3/assignment_loss‚1.3773596880159207 1loss/FBt3/assignment_loss‚0.48665176136\#fU]139743 .loss/FRt2/detection_loss‚2.006273979024792 %loss/total_loss‚48.31165001779124 /loss/FRt1/assignment_loss‚3.114036449030334 2loss/SRqqt1/assignment_loss‚1.4456868269953176 1loss/FBt2/assignment_loss‚0.28415613011023255 .loss/FRt1/detection_loss‚2.252363241186329 1loss/SRqqt1/detection_loss‚1.7084639163120934É(é¢óžÓ‚(:“ ³¨æÔ¦ŽôÕ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.559399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ù ¡þÔ‚(:ó ³¨æÔÀÚ# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 network.recv‚745240 disk.sda.out‚76.03125 cpu‚25.381671456756592 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚74.48622641576672 disk.nvme0c0n1.in‚162.9375 disk.dm-0.out‚ 7667.09375 disk.dm-1.out‚76.03125( proc.memory.availableMB‚ 953.24609375 disk.sda1.out‚0 disk.nvme0n1.in‚162.9375 disk.nvme0n1.out‚ 7667.09375$ memory_percent‚94.18184757232666 network.sent‚ 4.696049e+06 disk.rbd1.in‚0% disk./.usageGB‚1109.8986015319824 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚76.03125 disk.rbd2.out‚ 43.5205078125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 0.919921875 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.nvme0c0n1.out‚ 7667.09375 disk.dm-0.in‚162.9375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.in‚0QÎÒàžÕ‚(:“ ¨æÔÖ£©× gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.288799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷™ Ö‚(:ý ¨æÔí¨ïÙ disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.out‚ 76.66796875 disk.sda3.out‚ 76.66796875 disk.dm-0.in‚162.9375 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚94.18182373046875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0% disk./.usageGB‚1109.8986549377441 disk.rbd3.in‚0 disk.rbd2.out‚ 43.6455078125 disk.nvme0c0n1.in‚162.9375" proc.memory.availableMB‚953.25 network.recv‚751585 disk.nvme0n1.in‚162.9375 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 7667.83984375 network.sent‚ 4.804759e+06( disk./.usagePercent‚74.4862299998722" disk.nvme0n1.out‚ 7667.83984375 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 cpu‚25.37915696658226 disk.dm-0.out‚ 7667.83984375 disk.dm-1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 76.66796875n¹¯†‘ׂ(:† ѨæÔÏüùÕ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+a\iØ‚(:‡ ѨæÔÍ°íØ) disk./.usagePercent‚74.48622795181193$ disk./.usageGB‚1109.898624420166 disk.rbd1.in‚0 disk.sda3.out‚ 77.2578125 disk.rbd0.in‚0 disk.rbd2.out‚ 45.1728515625 disk.dm-0.in‚162.9375$ memory_percent‚94.18036937713623! disk.dm-0.out‚7765.4716796875 disk.rbd1.out‚ 0.919921875 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 77.2578125 disk.rbd3.in‚0 disk.sda2.out‚0 network.sent‚ 4.810332e+06 disk.nvme0c0n1.in‚162.9375$ disk.nvme0n1.out‚7765.4716796875 disk.drbd1777.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 953.48828125 cpu‚25.376703380112406 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚7765.4716796875 disk.sda.out‚ 77.2578125 disk.drbd1777.in‚0) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17 disk.sda3.in‚0# proc.memory.rssMB‚ 2875.26171875 network.recv‚755844 disk.sda.in‚0 disk.nvme0n1.in‚162.9375Ï)#Ù2Ù- _wandbruntime‚1966  _runtime‚1966FP¹ÉÚà _step‚202 ' lr-AdamW/pg1‚0.00017796689361890055 ' lr-AdamW/pg2‚0.00017796689361890055 trainer/global_step‚2499 % _timestamp‚1.7884498753739617e+09 _runtime‚1966.928381847~Ús”ÎÛÈ _step‚202 ' lr-AdamW/pg1‚0.00017796689361890055 ' lr-AdamW/pg2‚0.00017796689361890055 trainer/global_step‚2499 % _timestamp‚1.7884498753739617e+09 _runtime‚1966.928381847ÊS°ˆ2Ü- _wandbruntime‚1966  _runtime‚1966U®ÞØ'ÝÒO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36409152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚203 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2088018316669 _runtime‚1966.929198677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21245206 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36409154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47240979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161119950424064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158919817952768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36409152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21245204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143810387362821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141723023483904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145540093234688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12070986 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788449875374665e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33316192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143811058839933 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47220942 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2095735678464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 trainer/global_step‚2499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145540075231744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36409154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12070986 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788449875374665e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147635154548224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33316192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143811058839933 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47220942 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2095735678464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 trainer/global_step‚2499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145540075231744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33315736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36423170 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12075124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2096555119616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147692327014912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141777944653824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788449876043847e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 B8DeviceStat {UÝn sMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 1967.59884571 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158981468690432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158981492630528 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161182647244288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161182647244288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36423170 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47239396 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12075124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147693001393152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2096555119616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147692327014912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141777944653824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2095898744320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788449876043847e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 1967.59884571 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158981468690432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0̳ZŒR2â- _wandbruntime‚1967  _runtime‚1967B{é\ãÖ /loss/FBt1/detection_loss‚3.4481739898522696 trainer/global_step‚2499 % _timestamp‚1.7884498760455863e+09 _runtime‚1967.599844392 2loss/SRqqt3/assignment_loss‚1.5603808032778592 0loss/FBt4/assignment_loss‚0.3485047363031369 1loss/SRqqt2/detection_loss‚1.1982801481032035 epoch‚2 %loss/total_loss‚49.91258869277396 0loss/FRt1/assignment_loss‚3.6317784984906516 0loss/FRt2/assignment_loss‚1.7767695815939653 1loss/SRqqt4/detection_loss‚2.7719633834702626 2loss/SRqqt2/assignment_loss‚1.4025542453246218 /loss/FRt4/detection_loss‚3.2224629339964492 /loss/FBt3/detection_loss‚2.4858241617679595 /loss/FRt4/assignment_loss‚4.890255243881889 1loss/FBt2/assignment_loss‚0.29141214908466173 /loss/FRt3/detection_loss‚2.2472630574514993 /loss/FBt2/detection_loss‚0.8398760119592579 .loss/FBt4/detection_loss‚3.087406566509834 2loss/SRqqt1/assignment_loss‚1.5443997958844358 1loss/SRqqt3/detection_loss‚1.7944664579744523 _step‚205 0loss/FRt3/assignment_loss‚3.9807538484272205 2loss/SRqqt4/assignment_loss‚2.5028093473187516 0loss/FBt1/assignment_loss‚0.5528986894205445 0loss/FBt3/assignment_loss‚0.4660962540656328 0loss/SRqqt1/detection_loss‚1.735921268842437 .loss/FRt1/detection_loss‚2.521788768115498 /loss/FRt2/detection_loss‚1.6105487516574692‰fŽaäÛ /loss/FBt1/detection_loss‚3.4481739898522696 trainer/global_step‚2499 % _timestamp‚1.7884498760455863e+09 _runtime‚1967.599844392 2loss/SRqqt3/assignment_loss‚1.5603808032778592 0loss/FBt4/assignment_loss‚0.3485047363031369 1loss/SRqqt2/detection_loss‚1.1982801481032035 epoch‚2 %loss/total_loss‚49.91258869277396 0loss/FRt1/assignment_loss‚3.6317784984906516 0loss/FRt2/assignment_loss‚1.7767695815939653 1loss/SRqqt4/detection_loss‚2.7719633834702626 2loss/SRqqt2/assignment_loss‚1.4025542453246218 /loss/FRt4/detection_loss‚3.2224629339964492 /loss/FBt3/detection_loss‚2.4858241617679595 /loss/FRt4/assignment_loss‚4.890255243881889 1loss/FBt2/assignment_loss‚0.29141214908466173 /loss/FRt3/detection_loss‚2.2472630574514993 /loss/FBt2/detection_loss‚0.8398760119592579 .loss/FBt4/detection_loss‚3.087406566509834 2loss/SRqqt1/assignment_loss‚1.5443997958844358 1loss/SRqqt3/detection_loss‚1.7944664579744523 _step‚205 0loss/FRt3/assignment_loss‚3.9807538484272205 2loss/SRqqt4/assignment_loss‚2.5028093473187516 0loss/FBt1/assignment_loss‚0.5528986894205445 0loss/FBt3/assignment_loss‚0.4660962540656328 0loss/SRqqt1/detection_loss‚1.735921268842437 .loss/FRt1/detection_loss‚2.521788768115498 /loss/FRt2/detection_loss‚1.6105487516574692ÍÄ"žå‚(:“ à¨æÔ°¸ÃÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.065800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J+æ‚(:Š à¨æÔ¢‘Ù& disk.nvme0c0n1.out‚7863.3388671875 network.sent‚ 4.816517e+06$ disk.nvme0n1.out‚7863.3388671875 disk.dm-1.in‚0 disk.dm-1.out‚ 77.69140625 disk.sda.in‚0 disk.sda.out‚ 77.69140625 disk.rbd2.in‚0 disk.rbd3.out‚0 cpu‚25.374253481501547 disk.nvme0c0n1.in‚162.9375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 disk.rbd1.out‚ 0.919921875 disk.sda1.out‚0 disk.sda3.out‚ 77.69140625 disk.drbd1777.in‚0! disk.dm-0.out‚7863.3388671875$ memory_percent‚94.18342113494873 network.recv‚760363% disk./.usageGB‚1109.8969497680664 disk.rbd2.out‚ 45.2353515625 disk.nvme0n1.in‚162.9375 disk.sda2.in‚0( proc.memory.availableMB‚ 952.98828125( disk./.usagePercent‚74.4861155645044 disk.sda3.in‚0 disk.dm-0.in‚162.9375 disk.rbd1.in‚0 disk.rbd3.in‚0:«3“ç‚(:ˆ ï¨æÔÄ×»× gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0– è‚(:ˆ ï¨æÔ×ÜìÙ disk.sda.out‚ 78.55078125$ memory_percent‚94.18215751647949) proc.memory.percent‚17.54920482635498# disk./.usageGB‚1109.89701461792 disk.dm-0.in‚162.9375 disk.dm-1.out‚ 78.55078125' proc.memory.availableMB‚ 953.1953125! disk.dm-0.out‚7863.5888671875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.371827138849504 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚766371 disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 45.2509765625 disk.nvme0c0n1.in‚162.9375& disk.nvme0c0n1.out‚7863.5888671875 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2875.26171875 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.48611991663248 disk.rbd1.in‚0 disk.nvme0n1.in‚162.9375$ disk.nvme0n1.out‚7863.5888671875 disk.sda3.out‚ 78.55078125 disk.rbd0.out‚0 network.sent‚ 4.924259e+06 disk.sda2.in‚0 disk.sda3.in‚0ußê2é- _wandbruntime‚2004  _runtime‚2004õôÉêà trainer/global_step‚2549 % _timestamp‚1.7884499127840724e+09 _runtime‚2004.338667412 _step‚206 ' lr-AdamW/pg1‚0.00017747391495524548 ' lr-AdamW/pg2‚0.00017747391495524548w×WºÎëÈ trainer/global_step‚2549 % _timestamp‚1.7884499127840724e+09 _runtime‚2004.338667412 _step‚206 ' lr-AdamW/pg1‚0.00017747391495524548 ' lr-AdamW/pg2‚0.00017747391495524548Îéˆt52ì- _wandbruntime‚2004  _runtime‚2004:X+¾Ù'íÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164245694842368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146596429230229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884499127851064e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2128853656077 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2243158733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162002354835968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148357745177088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2004.339802111 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21653606 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168Ž<}å' îØO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2243158733312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162002354835968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2136721484288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48146994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21653604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150494466661376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148357745177088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2004.339802111 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37109952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚Xš·432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144468247051264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21653606 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150493792296448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164245694842368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164245513569280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162002378776064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146596429230229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148357745177088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884499127851064e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2128853656077 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2136065122304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33949723 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576ÏMôm2ï- _wandbruntime‚2005  _runtime‚2005{~”Kâ&ðÜM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661772 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2005.042767941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37123968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48165448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37123970 $ _timestamp‚1.788449913487762e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2244184223232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2549 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37123970 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168ðÉÖ¾ç&ñáM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12301002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37123970 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661772 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2005.042767941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33962776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37123968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2136884550144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48165448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48165448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37123970 $ _timestamp‚1.788449913487762e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164308028282880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2244184223232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2549 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151175397119488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2177707290112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34584514 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2286242164736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149382471097637 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151175379116544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22062004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884499501922524e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49093083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2177707290112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49093083 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151175397119488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22062006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844Ø8»`Þ'€ØO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22062004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884499501922524e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2169688995485 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49093083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2177707290112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49093083 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151175397119488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22062006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151175397119488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2177707290112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34584514 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2286242164736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149382471097637 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151175379116544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414Ód¬>ˆ2- _wandbruntime‚2042  _runtime‚2042؃(˜ã&‚ÝM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165146542456832 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49111563 trainer/global_step‚2599 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167433671484928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2177870355968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526539 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2042.458596258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884499509033854e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153409602511360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2287105088000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147268340670464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49111563 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151231750158336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153410276889600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 I?DeviceStatsMonitor.on_train_batch_;HI%end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2177870355968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49111563 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151231750158336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153410276889600 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2177870355968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526539 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2178526731264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2042.458596258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884499509033854e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153409602511360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2287105088000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147268340670464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768Ôø5•W2„- _wandbruntime‚2042  _runtime‚2042 èzƒ[…Õ 0loss/FBt1/assignment_loss‚0.5923025247951349 /loss/FRt1/assignment_loss‚4.139556188242776 2loss/SRqqt1/assignment_loss‚1.6224500177041539 /loss/FRt1/detection_loss‚2.8527288470949443 /loss/FBt2/detection_loss‚0.6377264128583496 %loss/total_loss‚50.76841822066416 _runtime‚2042.459782223 2loss/SRqqt2/assignment_loss‚1.1787349582235847 1loss/FBt2/assignment_loss‚0.33347409273884343 1loss/FBt3/assignment_loss‚0.49727728962898254 /loss/FBt1/detection_loss‚2.6385819911956787 /loss/FBt3/detection_loss‚3.0180854126811028 trainer/global_step‚2599 % _timestamp‚1.7884499509053814e+09 0loss/FRt2/assignment_loss‚1.4144276243944962 0loss/FRt3/assignment_loss‚3.7414598720414296 .loss/FRt3/detection_loss‚2.185591378116182 0loss/SRqqt1/detection_loss‚1.691773663151939 1loss/SRqqt4/detection_loss‚2.4723793864250183 _step‚213 /loss/FRt4/assignment_loss‚5.393703627586365 2loss/SRqqt3/assignment_loss‚1.4918597630057178 2loss/SRqqt4/assignment_loss‚2.3204368301800318 0loss/FBt4/assignment_loss‚0.7389580171762241 /loss/FRt2/detection_loss‚2.1434943499043584 -loss/FRt4/detection_loss‚3.46689196228981 1loss/SRqqt2/detection_loss‚1.1251872341252036 1loss/SRqqt3/detection_loss‚1.6248059912783201 /loss/FBt4/detection_loss‚3.4465307858255176 epoch‚2XgL`†Ú 0loss/SRqqt1/detection_loss‚1.691773663151939 1loss/SRqqt4/detection_loss‚2.4723793864250183 _step‚213 /loss/FRt4/assignment_loss‚5.393703627586365 2loss/SRqqt3/assignment_loss‚1.4918597630057178 2loss/SRqqt4/assignment_loss‚2.3204368301800318 0loss/FBt4/assignment_loss‚0.7389580171762241 /loss/FRt2/detection_loss‚2.1434943499043584 -loss/FRt4/detection_loss‚3.46689196228981 1loss/SRqqt2/detection_loss‚1.1251872341252036 1loss/SRqqt3/detection_loss‚1.6248059912783201 /loss/FBt4/detection_loss‚3.4465307858255176 epoch‚2 0loss/FBt1/assignment_loss‚0.5923025247951349 /loss/FRt1/assignment_loss‚4.139556188242776 2loss/SRqqt1/assignment_loss‚1.6224500177041539 /loss/FRt1/detection_loss‚2.8527288470949443 /loss/FBt2/detection_loss‚0.6377264128583496 %loss/total_loss‚50.76841822066416 _runtime‚2042.459782223 2loss/SRqqt2/assignment_loss‚1.1787349582235847 1loss/FBt2/assignment_loss‚0.33347409273884343 1loss/FBt3/assignment_loss‚0.49727728962898254 /loss/FBt1/detection_loss‚2.6385819911956787 /loss/FBt3/detection_loss‚3.0180854126811028 trainer/global_step‚2599 % _timestamp‚1.7884499509053814e+09 0loss/FRt2/assignment_loss‚1.4144276243944962 0loss/FRt3/assignment_loss‚3.7414598720414296 .loss/FRt3/detection_loss‚2.185591378116182Õb8Æ5ž‡‚(:“ «©æÔÅòÒÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.239' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.447800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂZW!ˆ‚(:– «©æÔ¼¨Ø( proc.memory.availableMB‚ 478.87890625) proc.memory.percent‚17.54920482635498 network.sent‚ 5.05023e+06" disk.dm-0.out‚8062.64404296875 disk.sda1.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚8062.64404296875 disk.sda1.in‚0 network.recv‚785312 disk.rbd2.out‚47.01708984375" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd3.in‚0) disk./.usagePercent‚74.48627608122837 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 80.5859375 disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 80.5859375 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚1109.899341583252' disk.nvme0c0n1.out‚8062.64404296875 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0$ memory_percent‚97.07715511322021 cpu‚25.362590106336956 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 80.5859375# proc.memory.rssMB‚ 2875.26171875?»#ž‰‚(:“ º©æÔùœýÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.933600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N Ø‘Š‚(:Ž º©æÔ†èïØ disk.drbd1777.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 478.578125) disk./.usagePercent‚74.48647653512768 disk.rbd3.in‚0 disk.sda.out‚ 81.05859375 disk.sda1.out‚0 cpu‚25.360453107285707 network.sent‚ 5.158934e+06 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 network.recv‚791478 disk.rbd3.out‚0 disk.sda3.out‚ 81.05859375 disk.rbd0.in‚0$ disk./.usageGB‚1109.902328491211" disk.nvme0n1.out‚ 8163.87109375 disk.dm-0.out‚ 8163.87109375 disk.sda2.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.54920482635498" disk.nvme0c0n1.in‚ 162.94140625$ disk.nvme0c0n1.out‚ 8163.87109375 disk.drbd1777.out‚0 disk.sda3.in‚0# memory_percent‚97.0789909362793 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 81.05859375 disk.sda1.in‚0# proc.memory.rssMB‚ 2875.26171875 disk.rbd2.out‚47.14208984375 disk.rbd0.out‚0 proc.cpu.threads‚17Í ‹2‹- _wandbruntime‚2079  _runtime‚2079ò.ôÇŒÁ & lr-AdamW/pg2‚0.0001764347150566883 trainer/global_step‚2649 % _timestamp‚1.7884499875676491e+09 _runtime‚2079.122005205 _step‚214 & lr-AdamW/pg1‚0.0001764347150566883ÓŠâÌÆ & lr-AdamW/pg2‚0.0001764347150566883 trainer/global_step‚2649 % _timestamp‚1.7884499875676491e+09 _runtime‚2079.122005205 _step‚214 & lr-AdamW/pg1‚0.0001764347150566883ÖQ“ Æ2Ž- _wandbruntime‚2079  _runtime‚2079v:okÙ'ÓO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22470406 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170496528439296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153993049061888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38511554 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49999098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2329075896832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚2079.122967488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12750208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170496349263360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88510650 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168167452542464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153993049061888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49999098 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209869897141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 % _timestamp‚1.7884499875683653e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35220176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153993049061888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49999098 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209869897141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149958643067904 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 % _timestamp‚1.7884499875683653e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35220176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49999098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2329075896832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚2079.122967488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12750208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153993031058944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170496349263360×Éãå2‘- _wandbruntime‚2079  _runtime‚2079Óªã&’ÝM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154049384097792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50037615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884499882391324e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156268240259584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22478572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150013564237824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50037615 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2079.794182905 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605Q1ßàè&“âM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2079.794182905 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38525570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154049384097792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50037615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884499882391324e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154049402100736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156268240259584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22478572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156268914637824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150013564237824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50037615 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12976057 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 TGDeviceStatsMonitor.on_train_bat£s´ëÔ!ch_start/inactive_split.large_pool.freed‚22878804 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173621759115264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39212352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12975635 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884500251229732e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156810683001344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39212354 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50925150 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2259022539776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154954554832453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚219 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944Wmh^Þ'¢ØO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2371955156992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154955226309565 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39212352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39212354 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50925150 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2259022539776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154954554832453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚219 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90137502 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152703849595904 _runtime‚2116.677647015 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 trainer/global_step‚2699 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35854439 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12976057 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22878804 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173621759115264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39212352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12975635 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884500251229732e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156810683001344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 % _timestamp‚1.7884500258250685e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚220 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2699 _runtime‚2117.380133681 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2252176109473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159127552386048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173684451751424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136¤\l}¥âM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2252176109473 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159127552386048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156867036040192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173684451751424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 % _timestamp‚1.7884500258250685e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_poJ§ŒÒol.freed‚50943604 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚220 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚2699 _runtime‚2117.380133681 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297Ü#ŠÅ2¦- _wandbruntime‚2117  _runtime‚2117mY 6Q§Ë 1loss/SRqqt1/detection_loss‚2.0803607534617186 0loss/SRqqt3/detection_loss‚1.827625942117763 0loss/SRqqt4/detection_loss‚2.756167617337457 /loss/FRt2/assignment_loss‚2.079721038303678 /loss/FRt4/assignment_loss‚5.547922580138497 1loss/FBt2/assignment_loss‚0.28522643492068284 0loss/FBt4/assignment_loss‚0.4038078940780674 .loss/FRt4/detection_loss‚3.273915321930595 0loss/FBt3/assignment_loss‚0.7025119744503172 .loss/FRt3/detection_loss‚2.043927038257772 /loss/FRt1/assignment_loss‚3.824416678126265 %loss/total_loss‚49.72026851813938 % _timestamp‚1.7884500258272104e+09 1loss/SRqqt2/assignment_loss‚1.377056925334561 /loss/FRt1/detection_loss‚2.4067789170800187 .loss/FBt3/detection_loss‚2.197628816589713 _runtime‚2117.381625913 _step‚221 1loss/SRqqt2/detection_loss‚1.5083252942897905 .loss/FBt1/detection_loss‚2.694856211543083 trainer/global_step‚2699 /loss/FRt2/detection_loss‚1.9545519075223379 2loss/SRqqt3/assignment_loss‚1.4899608242624212 0loss/SRqqt4/assignment_loss‚2.33099352491313 /loss/FRt3/assignment_loss‚3.016975450515747 0loss/SRqqt1/assignment_loss‚1.78565131733194 1loss/FBt1/assignment_loss‚0.40636289585381746 /loss/FBt2/detection_loss‚1.1787266948599648 .loss/FBt4/detection_loss‚2.546796464920044 epoch‚2¼’RV¨Ð 1loss/FBt1/assignment_loss‚0.40636289585381746 /loss/FBt2/detection_loss‚1.1787266948599648 .loss/FBt4/detection_loss‚2.546796464920044 epoch‚2 1loss/SRqqt1/detection_loss‚2.0803607534617186 0loss/SRqqt3/detection_loss‚1.827625942117763 0loss/SRqqt4/detection_loss‚2.756167617337457 /loss/FRt2/assignment_loss‚2.079721038303678 /loss/FRt4/assignment_loss‚5.547922580138497 1loss/FBt2/assignment_loss‚0.28522643492068284 0loss/FBt4/assignment_loss‚0.4038078940780674 .loss/FRt4/detection_loss‚3.273915321930595 0loss/FBt3/assignment_loss‚0.7025119744503172 .loss/FRt3/detection_loss‚2.043927038257772 /loss/FRt1/assignment_loss‚3.824416678126265 %loss/total_loss‚49.72026851813938 % _timestamp‚1.7884500258272104e+09 1loss/SRqqt2/assignment_loss‚1.377056925334561 /loss/FRt1/detection_loss‚2.4067789170800187 .loss/FBt3/detection_loss‚2.197628816589713 _runtime‚2117.381625913 _step‚221 1loss/SRqqt2/detection_loss‚1.5083252942897905 .loss/FBt1/detection_loss‚2.694856211543083 trainer/global_step‚2699 /loss/FRt2/detection_loss‚1.9545519075223379 2loss/SRqqt3/assignment_loss‚1.4899608242624212 0loss/SRqqt4/assignment_loss‚2.33099352491313 /loss/FRt3/assignment_loss‚3.016975450515747 0loss/SRqqt1/assignment_loss‚1.78565131733194ݶŠhšž©‚(:“ ö©æÔµ–ÌÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚147.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªojmª‚(:“ ö©æÔÛ¤§Ø$ disk./.usageGB‚1109.902919769287 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 83.1640625 disk.sda1.out‚0 cpu‚25.351941851404597 network.recv‚809170 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.48651621629548 disk.rbd2.out‚48.92041015625 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0! disk.dm-0.out‚8360.1884765625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875$ memory_percent‚97.07951545715332 network.sent‚ 5.283572e+06" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚8360.1884765625 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2875.26171875 proc.cpu.threads‚17$ disk.nvme0n1.out‚8360.1884765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 83.1640625' proc.memory.availableMB‚ 478.4921875) proc.memory.percent‚17.54920482635498 disk.dm-1.in‚0 disk.sda3.out‚ 83.1640625Cå,q“«‚(:ˆ …ªæÔÏé¬× gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.641' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.3282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¶©$¬‚(:™ …ªæÔöàÐÙ disk.rbd1.in‚0 disk.sda1.in‚0$ memory_percent‚97.07822799682617) disk./.usagePercent‚74.48651749633315% disk./.usageGB‚1109.9029388427734 disk.dm-1.out‚ 83.72265625" disk.dm-0.out‚8360.53369140625 network.recv‚815409 disk.rbd1.out‚ 0.919921875 disk.rbd3.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.in‚0 network.sent‚ 5.391748e+06 disk.rbd3.out‚0 disk.nvme0n1.in‚ 162.94140625 disk.rbd2.out‚48.93603515625 cpu‚25.349823547826908 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8360.53369140625 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625) proc.memory.percent‚17.54920482635498 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 83.72265625 disk.rbd2.in‚0% disk.nvme0n1.out‚8360.53369140625# proc.memory.rssMB‚ 2875.26171875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 83.72265625 disk.drbd1777.out‚0& proc.memory.availableMB‚ 478.703125¶~o2­- _wandbruntime‚2153  _runtime‚2153Çɮà ' lr-AdamW/pg1‚0.00017532525307213776 ' lr-AdamW/pg2‚0.00017532525307213776 trainer/global_step‚2749 % _timestamp‚1.7884500623785028e+09 _runtime‚2153.932998926 _step‚222OÊ0;Î¯È trainer/global_step‚2749 % _timestamp‚1.7884500623785028e+09 _runtime‚2153.932998926 _step‚222 ' lr-AdamW/pg1‚0.00017532525307213776 ' lr-AdamW/pg2‚0.00017532525307213776ÞIVœ÷2°- _wandbruntime‚2153  _runtime‚2153ÿìN[×'±ÑO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174332526308864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155449056123904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36489435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176747168860160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2292195013709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2300008345600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23287204 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155449073163264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161928343289344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159628352946688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51851202 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39913154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157740596699861 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51871239 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2300664707584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39913152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176747369007616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788450062379395e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2749 _step‚223 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174332502368768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚200373…=®'²ÖO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161928343289344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159628352946688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51851202 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39913154 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157740596699861 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51871239 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161929017654272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2300664707584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39913152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176747369007616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788450062379395e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2300664707584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2749 _step‚223 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36489860 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174332502368768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39913152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174332526308864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155449056123904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36489435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13202231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176747168860160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2292195013709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2300008345600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23287204 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155449073163264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13202654 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batchc¤³._start/segment.small_pool.allocated‚414ß %Â2³- _wandbruntime‚2154  _runtime‚2154ø³a2ã&´ÝM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157796988742705 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155503960254464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2292357010921 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51869656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155503977293824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91796824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155503960254464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2292357010921 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23295374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51869656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605à‘‹Ž2¶- _wandbruntime‚2154  _runtime‚2154„ÕK>W·Ñ 0loss/FBt3/assignment_loss‚0.5313475049204297 /loss/FRt2/detection_loss‚2.0461173420589343 _step‚225 /loss/FRt2/assignment_loss‚2.009704870336196 1loss/FBt4/assignment_loss‚0.43114226932213207 1loss/SRqqt2/detection_loss‚1.0977007172098665 -loss/FBt4/detection_loss‚2.96354873975118 /loss/FRt1/assignment_loss‚4.347843827385652 2loss/SRqqt4/assignment_loss‚2.2836474385754815 1loss/SRqqt4/detection_loss‚2.4297935757143745 .loss/FBt1/detection_loss‚3.371280536055565 % _timestamp‚1.7884500630658932e+09 _runtime‚2154.620208719 0loss/FRt3/assignment_loss‚3.6290421038866043 0loss/FBt1/assignment_loss‚0.3834143899504852 .loss/FRt1/detection_loss‚2.466387215805681 1loss/SRqqt3/detection_loss‚1.7291260991990567 /loss/FBt2/detection_loss‚0.9457047903475851 %loss/total_loss‚49.85616994345155 2loss/SRqqt2/assignment_loss‚1.0913175281814553 1loss/FBt2/assignment_loss‚0.19743517166258198 .loss/FRt4/assignment_loss‚5.07057892043015 2loss/SRqqt3/assignment_loss‚1.5131391869485378 trainer/global_step‚2749 2loss/SRqqt1/assignment_loss‚1.7328345499211741 /loss/FRt3/detection_loss‚2.1074024073779585 0loss/SRqqt1/detection_loss‚1.758143340387652 /loss/FRt4/detection_loss‚3.0431078158575913 .loss/FBt3/detection_loss‚2.676409602165222 epoch‚2j1\¸Ö /loss/FRt1/assignment_loss‚4.347843827385652 2loss/SRqqt4/assignment_loss‚2.2836474385754815 1loss/SRqqt4/detection_loss‚2.4297935757143745 .loss/FBt1/detection_loss‚3.371280536055565 % _timestamp‚1.7884500630658932e+09 _runtime‚2154.620208719 0loss/FRt3/assignment_loss‚3.6290421038866043 0loss/FBt1/assignment_loss‚0.3834143899504852 .loss/FRt1/detection_loss‚2.466387215805681 1loss/SRqqt3/detection_loss‚1.7291260991990567 /loss/FBt2/detection_loss‚0.9457047903475851 %loss/total_loss‚49.85616994345155 2loss/SRqqt2/assignment_loss‚1.0913175281814553 1loss/FBt2/assignment_loss‚0.19743517166258198 .loss/FRt4/assignment_loss‚5.07057892043015 2loss/SRqqt3/assignment_loss‚1.5131391869485378 trainer/global_step‚2749 2loss/SRqqt1/assignment_loss‚1.7328345499211741 /loss/FRt3/detection_loss‚2.1074024073779585 0loss/SRqqt1/detection_loss‚1.758143340387652 /loss/FRt4/detection_loss‚3.0431078158575913 .loss/FBt3/detection_loss‚2.676409602165222 epoch‚2 0loss/FBt3/assignment_loss‚0.5313475049204297 /loss/FRt2/detection_loss‚2.0461173420589343 _step‚225 /loss/FRt2/assignment_loss‚2.009704870336196 1loss/FBt4/assignment_loss‚0.43114226932213207 1loss/SRqqt2/detection_loss‚1.0977007172098665 -loss/FBt4/detection_loss‚2.96354873975118áÕËŠm’¹‚(:‡ ”ªæÔ•Ó‘Ö gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.475' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†@iù!º‚(:– ”ªæÔÔ˜ÃÙ! disk.dm-0.out‚8458.2021484375) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 84.49609375 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.068359375& disk.nvme0c0n1.out‚8458.2021484375$ disk.nvme0n1.out‚8458.2021484375$ memory_percent‚97.08142280578613 cpu‚25.347653825930802 disk.sda3.out‚ 84.49609375 network.sent‚ 5.398003e+06 disk.rbd0.in‚0 disk.rbd2.out‚48.99853515625 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 478.1796875 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.in‚0) disk./.usagePercent‚74.48651621629548 disk.sda3.in‚0 disk.dm-1.out‚ 84.49609375 disk.sda1.out‚0 network.recv‚819855$ disk./.usageGB‚1109.902919769287" disk.nvme0c0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2875.26171875Aq¦,’»‚(:‡ £ªæÔ׸éÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¿ÔÚ"¼‚(:— £ªæÔÔÐØ network.sent‚ 5.505748e+06! disk.dm-0.out‚8555.6279296875 disk.rbd1.out‚ 1.068359375 disk.sda1.out‚0 network.recv‚825860) disk./.usagePercent‚74.48651749633315% disk./.usageGB‚1109.9029388427734 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 85.06640625 disk.rbd0.out‚0 disk.rbd2.out‚48.99853515625$ disk.nvme0n1.out‚8555.6279296875 disk.dm-0.in‚ 162.94140625 disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.345646484960096 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 85.06640625 disk.nvme0n1.in‚ 162.94140625 disk.sda2.out‚0' proc.memory.availableMB‚ 478.6640625) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 85.06640625$ memory_percent‚97.07846641540527" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚8555.6279296875 disk.rbd3.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2875.26171875&†îšœ½‚(:‘ ²ªæÔ‡ÅÁÕ gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.4' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.880000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wgñ°¾‚(:’ ²ªæÔ¿õÿØ) disk./.usagePercent‚74.48651621629548 disk.rbd3.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 network.sent‚ 5.51214e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚8556.04345703125% disk.nvme0n1.out‚8556.04345703125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 478.41796875 cpu‚25.34363616240518 disk.dm-1.in‚0 disk.dm-1.out‚ 85.828125 disk.rbd1.out‚ 1.1005859375 disk.rbd2.out‚ 49.1591796875 network.recv‚830327 disk.sda3.in‚0 disk.sda3.out‚ 85.828125 disk.rbd2.in‚0$ memory_percent‚97.07996845245361 proc.cpu.threads‚17$ disk./.usageGB‚1109.902919769287 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚8556.04345703125 disk.sda.out‚ 85.828125 disk.nvme0n1.in‚ 162.94140625¤á2¿- _wandbruntime‚2190  _runtime‚2190ÂÔ…°ÉÀà ' lr-AdamW/pg1‚0.00017474446052829964 ' lr-AdamW/pg2‚0.00017474446052829964 trainer/global_step‚2799 % _timestamp‚1.7884500987102463e+09 _runtime‚2190.264689927 _step‚226 }Ÿ«ÎÁÈ % _timestamp‚1.7884500987102463e+09 _runtime‚2190.264689927 _step‚226 ' lr-AdamW/pg1‚0.00017474446052829964 ' lr-AdamW/pg2‚0.00017474446052829964 trainer/global_step‚2799ânIžâ2Â- _wandbruntime‚2190  _runtime‚2190‹ÅŠWÃÓO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179872819127296 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40613954 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341650513408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23695604 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162446004889088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52777254 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177415063192064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2340994151424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13430136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚227 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40613952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158194262651904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2333030353117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 _runtime‚2190.265607065 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40613952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 % _timestamp‚1.7884500987110288e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 % _timestamp‚1.7884500987110288e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341650513408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23695604 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162446004889088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52777254 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177415063192064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52777254 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2340994151424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13430136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚227 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚2799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40613952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037㕈ôø2Å- _wandbruntime‚2190  _runtime‚2190ß%œ”â&ÆÜM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884500993831847e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚2799 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177476689989632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93443741 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 _step‚228 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703772 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158249166782464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52795708 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93443741 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40627970 _step‚228 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179935134786048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703772 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158249166782464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52795708 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_splitG.ñ8~ .all.current‚453 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884500993831847e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚2799 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430ä|¸†2È- _wandbruntime‚2190  _runtime‚2190Nº,nTÉÎ 1loss/SRqqt1/detection_loss‚1.8480973845382906 .loss/FBt1/detection_loss‚3.511269768079122 epoch‚2 % _timestamp‚1.7884500993849752e+09 -loss/FRt3/assignment_loss‚3.2128101322386 2loss/SRqqt4/assignment_loss‚2.3495919555425644 /loss/FBt3/assignment_loss‚0.616233186589347 /loss/FRt3/detection_loss‚2.1153448494496168 trainer/global_step‚2799 _runtime‚2190.939239804 /loss/FRt2/assignment_loss‚1.647539338287042 1loss/SRqqt1/assignment_loss‚1.513937596512853 .loss/FRt4/detection_loss‚3.579666131734848 0loss/FBt4/assignment_loss‚0.6752722497098148 %loss/total_loss‚48.18839296921581 .loss/FRt2/detection_loss‚1.782363265448687 0loss/SRqqt2/detection_loss‚0.975993771920912 1loss/SRqqt3/detection_loss‚1.6246520947172958 -loss/FBt3/detection_loss‚2.33697180946668 0loss/FRt4/assignment_loss‚4.8386714339256285 2loss/SRqqt2/assignment_loss‚1.4510821657022461 2loss/SRqqt3/assignment_loss‚1.4626041738984157 0loss/FRt1/assignment_loss‚3.4545127043918686 1loss/FBt1/assignment_loss‚0.49964270802835625 1loss/FBt2/assignment_loss‚0.22696511328921742 0loss/SRqqt4/detection_loss‚2.550924281279246 /loss/FRt1/detection_loss‚2.3752912465406926 /loss/FBt2/detection_loss‚0.8653370111322483 /loss/FBt4/detection_loss‚2.6736185967922212 _step‚229=YÊÓ /loss/FRt1/detection_loss‚2.3752912465406926 /loss/FBt2/detection_loss‚0.8653370111322483 /loss/FBt4/detection_loss‚2.6736185967922212 _step‚229 1loss/SRqqt1/detection_loss‚1.8480973845382906 .loss/FBt1/detection_loss‚3.511269768079122 epoch‚2 % _timestamp‚1.7884500993849752e+09 -loss/FRt3/assignment_loss‚3.2128101322386 2loss/SRqqt4/assignment_loss‚2.3495919555425644 /loss/FBt3/assignment_loss‚0.616233186589347 /loss/FRt3/detection_loss‚2.1153448494496168 trainer/global_step‚2799 _runtime‚2190.939239804 /loss/FRt2/assignment_loss‚1.647539338287042 1loss/SRqqt1/assignment_loss‚1.513937596512853 .loss/FRt4/detection_loss‚3.579666131734848 0loss/FBt4/assignment_loss‚0.6752722497098148 %loss/total_loss‚48.18839296921581 .loss/FRt2/detection_loss‚1.782363265448687 0loss/SRqqt2/detection_loss‚0.975993771920912 1loss/SRqqt3/detection_loss‚1.6246520947172958 -loss/FBt3/detection_loss‚2.33697180946668 0loss/FRt4/assignment_loss‚4.8386714339256285 2loss/SRqqt2/assignment_loss‚1.4510821657022461 2loss/SRqqt3/assignment_loss‚1.4626041738984157 0loss/FRt1/assignment_loss‚3.4545127043918686 1loss/FBt1/assignment_loss‚0.49964270802835625 1loss/FBt2/assignment_loss‚0.22696511328921742 0loss/SRqqt4/detection_loss‚2.550924281279246å*Ýk“Ë‚(:ˆ ÁªæÔŸÍîÖ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úê§ßÌ‚(:” ÁªæÔŒ–ÊØ disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 86.6015625 disk.sda1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 478.58984375 disk.sda.in‚0 network.sent‚ 5.5178e+06 network.recv‚834723 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚8653.59814453125 disk.sda2.out‚0 disk.sda.out‚ 86.6015625 proc.cpu.threads‚17 disk.rbd2.out‚ 50.7333984375 disk.sda3.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.54920482635498 cpu‚25.34167132707763% disk.nvme0n1.out‚8653.59814453125 disk.dm-1.in‚0$ memory_percent‚97.07891941070557 disk.sda3.out‚ 86.6015625" disk.nvme0c0n1.in‚ 162.94140625' disk.nvme0c0n1.out‚8653.59814453125 disk.sda2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2875.26171875) disk./.usagePercent‚74.48651621629548$ disk./.usageGB‚1109.902919769287 disk.rbd1.out‚ 1.1005859375V8V؜͂(:‘ ЪæÔœàíÕ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.2' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.839999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v e΂(:” ЪæÔŒÛß× cpu‚25.339742986148966 network.recv‚841221% disk./.usageGB‚1109.9025764465332 disk.rbd1.out‚ 1.16357421875 disk.rbd2.out‚ 50.8740234375 disk.sda2.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 162.94140625# memory_percent‚97.0804214477539 disk.sda3.in‚0 disk.drbd1777.in‚0" disk.dm-0.out‚8654.28369140625 disk.sda2.in‚0 disk.dm-1.out‚ 86.8515625 disk.nvme0n1.in‚ 162.94140625% disk.nvme0n1.out‚8654.28369140625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 162.94140625' disk.nvme0c0n1.out‚8654.28369140625# proc.memory.rssMB‚ 2875.26171875 disk.rbd0.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 86.8515625( disk./.usagePercent‚74.4864931756174 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda.in‚0) proc.memory.percent‚17.54920482635498 network.sent‚ 5.626506e+06 disk.sda.out‚ 86.8515625 disk.rbd0.in‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 478.34375KÄ´w2Ï- _wandbruntime‚2226  _runtime‚2226ϬJ±ÇÐÁ _step‚230 & lr-AdamW/pg1‚0.0001741464537942687 & lr-AdamW/pg2‚0.0001741464537942687 trainer/global_step‚2849 % _timestamp‚1.7884501347357194e+09 _runtime‚2226.290130047©WrWÌÑÆ trainer/global_step‚2849 % _timestamp‚1.7884501347357194e+09 _runtime‚2226.290130047 _step‚230 & lr-AdamW/pg1‚0.0001741464537942687 & lr-AdamW/pg2‚0.0001741464537942687æ´ìG‘2Ò- _wandbruntime‚2226  _runtime‚22268®Ù'ÓÓO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24104004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37762034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165263638828544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2382636319232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884501347365174e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41314752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 trainer/global_step‚2849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160939486219264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180497600075264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2226.291136213 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167645618785792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53703306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2382636319232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658030 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872(Ñç#Þ'ÔØO _runtime‚2226.291136213 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167645618785792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53703306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2382636319232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658030 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24104004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37762034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165263638828544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2382636319232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884501347365174e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41314752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 trainer/global_step‚2849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160939486219264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180497600075264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0çO--‹2Õ- _wandbruntime‚2226  _runtime‚2226‰Õ¼hæ ÖÝM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53741823 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13662512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167703465630720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2849 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2226.964426615 _step‚232|n0è&×âM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53741823 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41328770 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13662512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167703465630720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2849 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚2226.964426615 _step‚232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240èÍÙaõ2Ø- _wandbruntime‚2226  _runtime‚2226˜d SÙÍ .loss/FBt4/detection_loss‚2.856196986304389 2loss/SRqqt4/assignment_loss‚2.4298146718314717 /loss/FRt3/detection_loss‚2.1775371996419772 /loss/FBt3/detection_loss‚2.3833721786737443 1loss/SRqqt2/assignment_loss‚1.544550392474677 0loss/SRqqt3/detection_loss‚1.778535822342182 0loss/FBt4/assignment_loss‚0.5577694674720988 -loss/FBt1/detection_loss‚5.12692666053772 $ _timestamp‚1.788450135411186e+09 _step‚233 1loss/FBt2/assignment_loss‚0.26686480056963224 .loss/FRt4/detection_loss‚3.265628299542836 epoch‚2 %loss/total_loss‚52.33297495742646 /loss/FRt3/assignment_loss‚4.203500811542783 1loss/SRqqt3/assignment_loss‚1.793631582681475 /loss/FRt2/detection_loss‚2.0540009359163895 1loss/SRqqt1/detection_loss‚1.6883010771430906 /loss/FBt2/detection_loss‚0.8288138344578541 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.897177260975505 0loss/FRt2/assignment_loss‚2.1051541850680398 1loss/FBt1/assignment_loss‚0.10417237134242896 -loss/FRt1/detection_loss‚2.37627844928309 1loss/SRqqt2/detection_loss‚1.1106056720018387 0loss/SRqqt4/detection_loss‚2.655499629676342 /loss/FRt4/assignment_loss‚5.047437582697187 2loss/SRqqt1/assignment_loss‚1.5119187596757881 0loss/FBt3/assignment_loss‚0.5692863255739212 _runtime‚2226.965536327Û XÚÒ 2loss/SRqqt4/assignment_loss‚2.4298146718314717 /loss/FRt3/detection_loss‚2.1775371996419772 /loss/FBt3/detection_loss‚2.3833721786737443 1loss/SRqqt2/assignment_loss‚1.544550392474677 0loss/SRqqt3/detection_loss‚1.778535822342182 0loss/FBt4/assignment_loss‚0.5577694674720988 -loss/FBt1/detection_loss‚5.12692666053772 $ _timestamp‚1.788450135411186e+09 _step‚233 1loss/FBt2/assignment_loss‚0.26686480056963224 .loss/FRt4/detection_loss‚3.265628299542836 epoch‚2 %loss/total_loss‚52.33297495742646 /loss/FRt3/assignment_loss‚4.203500811542783 1loss/SRqqt3/assignment_loss‚1.793631582681475 /loss/FRt2/detection_loss‚2.0540009359163895 1loss/SRqqt1/detection_loss‚1.6883010771430906 /loss/FBt2/detection_loss‚0.8288138344578541 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.897177260975505 0loss/FRt2/assignment_loss‚2.1051541850680398 1loss/FBt1/assignment_loss‚0.10417237134242896 -loss/FRt1/detection_loss‚2.37627844928309 1loss/SRqqt2/detection_loss‚1.1106056720018387 0loss/SRqqt4/detection_loss‚2.655499629676342 /loss/FRt4/assignment_loss‚5.047437582697187 2loss/SRqqt1/assignment_loss‚1.5119187596757881 0loss/FBt3/assignment_loss‚0.5692863255739212 _runtime‚2226.965536327 .loss/FBt4/detection_loss‚2.856196986304389é(UŠî’Û‚(:‡ ߪæÔ­ÅÞÕ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×3–Ü‚(:“ ߪæÔðÑÒØ disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 87.5859375 cpu‚25.337688595484277 disk.sda.in‚0 disk.dm-0.out‚8752.091796875 disk.dm-1.in‚0) proc.memory.percent‚17.54920482635498 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 162.94140625 network.recv‚844838 disk.rbd1.out‚ 1.16357421875 disk.sda1.in‚0# disk.nvme0n1.out‚8752.091796875 proc.cpu.threads‚17 disk.sda.out‚ 87.5859375 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625) disk./.usagePercent‚74.48655871354617 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 50.8740234375% disk.nvme0c0n1.out‚8752.091796875 network.sent‚ 5.631748e+06 disk.sda3.out‚ 87.5859375$ memory_percent‚97.07915782928467( proc.memory.availableMB‚ 478.55078125# proc.memory.rssMB‚ 2875.26171875% disk./.usageGB‚1109.9035530090332´Òe½žÝ‚(:“ îªæÔïÛÌÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.473799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~­Þ‚(:“ îªæÔœø‚Ø network.sent‚ 5.740535e+06 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd3.out‚0$ memory_percent‚97.07860946655273 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 88.1171875 disk.rbd1.in‚0& proc.memory.availableMB‚ 478.640625 proc.cpu.threads‚17 disk.drbd1777.in‚0" disk.dm-0.out‚8752.73681640625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 88.1171875# proc.memory.rssMB‚ 2875.26171875 network.recv‚851206 disk.rbd1.out‚ 1.2265625 disk.sda1.in‚0 disk.sda3.in‚0) disk./.usagePercent‚74.47679663425052% disk./.usageGB‚1109.7580909729004 disk.rbd0.out‚0 disk.rbd2.out‚50.97607421875' disk.nvme0c0n1.out‚8752.73681640625 disk.nvme0n1.in‚ 162.94140625 disk.sda.out‚ 88.1171875 disk.rbd0.in‚0% disk.nvme0n1.out‚8752.73681640625) proc.memory.percent‚17.54920482635498 cpu‚25.33580661066462ÉöÐ2ß- _wandbruntime‚2262  _runtime‚2262ú2ÚìÉàà ' lr-AdamW/pg2‚0.00017353135749375198 trainer/global_step‚2899 % _timestamp‚1.7884501707754588e+09 _runtime‚2262.329851186 _step‚234 ' lr-AdamW/pg1‚0.00017353135749375198þ/—ÎáÈ _step‚234 ' lr-AdamW/pg1‚0.00017353135749375198 ' lr-AdamW/pg2‚0.00017353135749375198 trainer/global_step‚2899 % _timestamp‚1.7884501707754588e+09 _runtime‚2262.329851186ê`ãÜ2â- _wandbruntime‚2262  _runtime‚2262;Pyàu&ãÓO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54629358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183580113018368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2414046594181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42015554 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2422965763072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚235 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512406 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 _runtime‚2262.331143962 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186123929905664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163684675707904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186124127955968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96664949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2899 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_spl }Õdit.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1660993937791975ãöÅÞ'äØO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2422965763072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13886135 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚235 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42015552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24512404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512406 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 _runtime‚2262.331143962 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54629358 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186123929905664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163684675707904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2414701031933 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186124127955968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96664949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2899 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168081308773888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2543990997504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166099393779197 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54629358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚183580113018368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183580136958464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2414046594181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38398539 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42015554 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174110208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872릪éÔ2å- _wandbruntime‚2263  _runtime‚2263f…pÙã&æÝM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170562103378944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54647812 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168137643809792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13890760 % _timestamp‚1.7884501714491322e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029570 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166155114344929 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163739596877824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186186451113472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424441566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2544687357440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183641763756032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166154442867609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2423785190912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2899 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186186451113472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424441566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2544687357440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183641763756032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166154442867609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2423785190912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2899 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173363568647168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24920806 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186662673841664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166429899275264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2586800247808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186662649901568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2455536371341 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚239 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2586972260864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884502068168457e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168885435646605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42716352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173362894282240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170898960716288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189249450149376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166429882235904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463951568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2455536371341 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚239 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2586972260864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884502068168457e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168885435646605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42716352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173362894282240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170898960716288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189249450149376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166429882235904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463951568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173363568647168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24920806 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186662673841664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166429899275264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2586800247808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186662649901568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42730370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186724324579328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173420741127168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189312147367936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170955313755136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39048820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 # _timestamp‚1.78845020748524e+09 _step‚240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚ 2299.04020858 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465427372032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470*(å&ùßM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170955313755136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39048820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 # _timestamp‚1.78845020748524e+09 _step‚240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚ 2299.04020858 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465427372032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42730368 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42730370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55593927 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173420066748928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42730370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186724324579328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173420741127168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189312147367936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56481462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2495717272997 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173716594655744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25329204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169175105803264 _runtime‚2334.128402852 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884502425738206e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171671477514013 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496371710749 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 _step‚243 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173716612658688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629864687104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901«tÝÛ †ØO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43417154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43417152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25329204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176222206395392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169175105803264 _runtime‚2334.128402852 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884502425738206e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_ba Äsætch_start/requested_bytes.all.allocated‚171671477514013 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496371710749 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 _step‚243 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99898614 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173716612658688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629864687104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56481462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56481462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2495717272997 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173716594655744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2999 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25337374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176278704497152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99951149 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43431170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2630728366080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ÷¢bè&‰âM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25337374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171727198079745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176278704497152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189806861462528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173772965697536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99951149 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192437589828608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43431170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2630728366080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2630562657792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚2334.789896189 _step‚244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2999ô ¸JU2Š- _wandbruntime‚2334  _runtime‚2334üxY‹Ó /loss/FRt3/assignment_loss‚3.832311820100855 1loss/SRqqt1/detection_loss‚1.8109510447140094 1loss/SRqqt2/detection_loss‚1.1996116782997694 1loss/SRqqt4/detection_loss‚2.5040755019043432 /loss/FBt2/detection_loss‚0.8852323523139168 _runtime‚2334.791252225 0loss/FBt3/assignment_loss‚0.4229949365369919 .loss/FRt4/detection_loss‚3.334464555978775 /loss/FBt1/detection_loss‚2.9145436882972717 2loss/SRqqt1/assignment_loss‚1.7353766292885497 /loss/FRt2/detection_loss‚1.7482475888729097 trainer/global_step‚2999 $ _timestamp‚1.788450243236659e+09 _step‚245 /loss/FRt4/assignment_loss‚4.766854250431061 /loss/FRt1/detection_loss‚2.4473901624615126 0loss/SRqqt3/detection_loss‚2.034376776948267 /loss/FRt2/assignment_loss‚1.811139430999756 %loss/total_loss‚48.59479468802773 2loss/SRqqt3/assignment_loss‚1.6827121127244768 1loss/FBt2/assignment_loss‚0.19386753230828782 0loss/FRt1/assignment_loss‚2.5595022423847302 2loss/SRqqt2/assignment_loss‚1.0882431251213323 2loss/SRqqt4/assignment_loss‚2.6565454828016684 0loss/FBt4/assignment_loss‚0.7268492048597182 /loss/FBt3/detection_loss‚2.7379098355770113 0loss/FBt1/assignment_loss‚0.7203410007059574 /loss/FRt3/detection_loss‚2.0147016363011465 .loss/FBt4/detection_loss‚2.766552098095417 epoch‚2ëm²^ŒØ %loss/total_loss‚48.59479468802773 2loss/SRqqt3/assignment_loss‚1.6827121127244768 1loss/FBt2/assignment_loss‚0.19386753230828782 0loss/FRt1/assignment_loss‚2.5595022423847302 2loss/SRqqt2/assignment_loss‚1.0882431251213323 2loss/SRqqt4/assignment_loss‚2.6565454828016684 0loss/FBt4/assignment_loss‚0.7268492048597182 /loss/FBt3/detection_loss‚2.7379098355770113 0loss/FBt1/assignment_loss‚0.7203410007059574 /loss/FRt3/detection_loss‚2.0147016363011465 .loss/FBt4/detection_loss‚2.766552098095417 epoch‚2 /loss/FRt3/assignment_loss‚3.832311820100855 1loss/SRqqt1/detection_loss‚1.8109510447140094 1loss/SRqqt2/detection_loss‚1.1996116782997694 1loss/SRqqt4/detection_loss‚2.5040755019043432 /loss/FBt2/detection_loss‚0.8852323523139168 _runtime‚2334.791252225 0loss/FBt3/assignment_loss‚0.4229949365369919 .loss/FRt4/detection_loss‚3.334464555978775 /loss/FBt1/detection_loss‚2.9145436882972717 2loss/SRqqt1/assignment_loss‚1.7353766292885497 /loss/FRt2/detection_loss‚1.7482475888729097 trainer/global_step‚2999 $ _timestamp‚1.788450243236659e+09 _step‚245 /loss/FRt4/assignment_loss‚4.766854250431061 /loss/FRt1/detection_loss‚2.4473901624615126 0loss/SRqqt3/detection_loss‚2.034376776948267 /loss/FRt2/assignment_loss‚1.811139430999756õì~H‘‚(:† È«æÔªºàÖ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.03' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëk-AŽ‚(:ˆ È«æÔ‰ÈµØ disk.sda2.in‚0 disk.rbd2.out‚ 52.99609375 disk.dm-1.out‚91.1875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 477.9921875# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 network.recv‚881051$ disk./.usageGB‚1109.756664276123 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0 cpu‚25.325030067853994 disk.sda3.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625% disk.nvme0n1.out‚9047.07275390625 disk.drbd1777.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚9047.07275390625 network.sent‚ 5.980712e+06 disk.sda1.in‚0' disk.nvme0c0n1.out‚9047.07275390625 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚97.08256721496582 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚91.1875) disk./.usagePercent‚74.47670088743271 disk.sda3.out‚91.1875 disk.rbd1.out‚ 1.2265625a,%“‚(:ˆ ׫æÔ‰ôÎÕ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûUJ’“‚(:† ׫æԓϔ٠network.recv‚886916 disk.rbd2.in‚0 disk.rbd2.out‚ 52.99609375% disk.nvme0c0n1.out‚9144.533203125 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0! disk./.usageGB‚1109.755859375 disk.rbd0.in‚0) proc.memory.percent‚17.54920482635498' disk./.usagePercent‚74.476646869843 disk.rbd3.out‚0 disk.rbd0.out‚0 network.sent‚ 6.088463e+06 disk.rbd3.in‚0 disk.sda.out‚ 91.9765625 disk.sda2.in‚0 disk.dm-1.out‚ 91.9765625$ memory_percent‚97.08163738250732 disk.rbd1.out‚ 1.2265625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.out‚9144.533203125( proc.memory.availableMB‚ 478.14453125# proc.memory.rssMB‚ 2875.26171875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 91.9765625 disk.nvme0n1.in‚–ïšç~ 162.94140625# disk.nvme0n1.out‚9144.533203125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.32328095789078kw?2‘- _wandbruntime‚2369  _runtime‚2369_Mg-É’Ã ' lr-AdamW/pg2‚0.00017158483069094042 trainer/global_step‚3049 % _timestamp‚1.7884502782969844e+09 _runtime‚2369.851369775 _step‚246 ' lr-AdamW/pg1‚0.00017158483069094042éªBÎ“È ' lr-AdamW/pg2‚0.00017158483069094042 trainer/global_step‚3049 % _timestamp‚1.7884502782969844e+09 _runtime‚2369.851369775 _step‚246 ' lr-AdamW/pg1‚0.00017158483069094042ö÷î”Á2”- _wandbruntime‚2369  _runtime‚2369Í—YÝ'•×O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2672773139968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179080844143616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57427551 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176534246598144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¯µâ'–ÜO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884502782977283e+09 _runtime‚2369.852390929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44117952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25737606 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179080844143616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57427551 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176534246598144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2672773139968 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2547398983680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 % _timestamp‚1.7884502789624474e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2547398983680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 % _timestamp‚1.7884502789624474e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 F³2£- _wandbruntime‚2405  _runtime‚2405}ßU¬Ç¤Á # _timestamp‚1.78845031400385e+09 _runtime‚2405.558202599 _step‚250 ' lr-AdamW/pg1‚0.00017090269318523416 ' lr-AdamW/pg2‚0.00017090269318523416 trainer/global_step‚3099:PZìÌ¥Æ # _timestamp‚1.78845031400385e+09 _runtime‚2405.558202599 _step‚250 ' lr-AdamW/pg1‚0.00017090269318523416 ' lr-AdamW/pg2‚0.00017090269318523416 trainer/global_step‚3099ú/•l2¦- _wandbruntime‚2405  _runtime‚2405†'KfÝ'§×O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 % _timestamp‚1.7884503140045638e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58333566 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26146004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚3099 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2587565348352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174665501819904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2587565348352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195910260551168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578042389565 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181939481891840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179351898540544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181938807526912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2586908986368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177243561248829 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179351898540544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626230079488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2587565348352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174665501819904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2587565348352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195910260551168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578042389565 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179351916543488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.o `™ón_train_batch_start/active_bytes.all.allocated‚181939481891840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179351898540544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181938807526912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2586908986368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177243561248829 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179351898540544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626230079488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 % _timestamp‚1.7884503140045638e+09û†Ÿ¸á2©- _wandbruntime‚2406  _runtime‚2406@Cñ¡ç&ªáM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578204386777 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58352020 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716985193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 trainer/global_step‚3099 % _timestamp‚1.7884503146680927e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58372083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58352020 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179408269582336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58372083 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195971911288832 _step‚252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2406.222891109 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2578858824737 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179408251579392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181995979993600:æô+ì&«æM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14803277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716985193984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036992 trainer/global_step‚3099 % _timestamp‚1.7884503146680927e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26154174 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58372083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58352020 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14803732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44832770 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44832770 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179408269582336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58372083 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195971911288832 _step‚252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44832768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2406.222891109 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177299281814561 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2578858824737 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179408251579392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181995979993600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578204386777 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716811097088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58352020 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455ü>2¬- _wandbruntime‚2406  _runtime‚2406GVV­Ð 1loss/SRqqt4/assignment_loss‚2.276952311396599 /loss/FBt3/assignment_loss‚0.790069505572319 /loss/FRt2/detection_loss‚1.6643791467599247 %loss/total_loss‚48.22471475114461 epoch‚2 /loss/FRt2/assignment_loss‚1.705143796361011 .loss/FRt4/assignment_loss‚5.65982605792858 2loss/SRqqt1/assignment_loss‚1.2057640653241564 /loss/FRt1/assignment_loss‚2.807552659286643 1loss/SRqqt2/detection_loss‚1.5152441612919982 1loss/SRqqt4/detection_loss‚2.6648083925247192 /loss/FBt3/detection_loss‚2.9505352899432182 trainer/global_step‚3099 _step‚253 2loss/SRqqt2/assignment_loss‚1.2729916935216408 0loss/FBt4/assignment_loss‚0.5329103493477305 % _timestamp‚1.7884503146696336e+09 1loss/FBt1/assignment_loss‚0.48179953543535703 /loss/FRt3/detection_loss‚2.1238781399322004 .loss/FRt4/detection_loss‚3.124331032788312 /loss/FBt1/detection_loss‚3.2768619284033775 .loss/FBt4/detection_loss‚2.854158954694867 1loss/FBt2/assignment_loss‚0.22028564811429102 .loss/FRt1/detection_loss‚2.137848387911635 /loss/FBt2/detection_loss‚0.8705215955179234 /loss/FRt3/assignment_loss‚3.231021182717017 1loss/SRqqt3/assignment_loss‚1.530815018012243 1loss/SRqqt1/detection_loss‚1.6843530938581184 1loss/SRqqt3/detection_loss‚1.6426628045007294 _runtime‚2406.223969231)Eâr[®Õ /loss/FBt3/assignment_loss‚0.790069505572319 /loss/FRt2/detection_loss‚1.6643791467599247 %loss/total_loss‚48.22471475114461 epoch‚2 /loss/FRt2/assignment_loss‚1.705143796361011 .loss/FRt4/assignment_loss‚5.65982605792858 2loss/SRqqt1/assignment_loss‚1.2057640653241564 /loss/FRt1/assignment_loss‚2.807552659286643 1loss/SRqqt2/detection_loss‚1.5152441612919982 1loss/SRqqt4/detection_loss‚2.6648083925247192 /loss/FBt3/detection_loss‚2.9505352899432182 trainer/global_step‚3099 _step‚253 2loss/SRqqt2/assignment_loss‚1.2729916935216408 0loss/FBt4/assignment_loss‚0.5329103493477305 % _timestamp‚1.7884503146696336e+09 1loss/FBt1/assignment_loss‚0.48179953543535703 /loss/FRt3/detection_loss‚2.1238781399322004 .loss/FRt4/detection_loss‚3.124331032788312 /loss/FBt1/detection_loss‚3.2768619284033775 .loss/FBt4/detection_loss‚2.854158954694867 1loss/FBt2/assignment_loss‚0.22028564811429102 .loss/FRt1/detection_loss‚2.137848387911635 /loss/FBt2/detection_loss‚0.8705215955179234 /loss/FRt3/assignment_loss‚3.231021182717017 1loss/SRqqt3/assignment_loss‚1.530815018012243 1loss/SRqqt1/detection_loss‚1.6843530938581184 1loss/SRqqt3/detection_loss‚1.6426628045007294 _runtime‚2406.223969231 1loss/SRqqt4/assignment_loss‚2.276952311396599ýPzÇKž¯‚(:“ “¬æÔ×ÁÚÕ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.942' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.188399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Эí1°‚(:¦ “¬æÔŸ¸ï× disk.rbd2.in‚0% disk.nvme0c0n1.out‚9341.212890625# disk.nvme0n1.out‚9341.212890625" disk.nvme0c0n1.in‚ 162.94140625 network.sent‚ 6.294654e+06 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.316487914610438% disk./.usageGB‚1109.7569122314453 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 94.51171875 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0$ memory_percent‚97.08366394042969 disk.rbd0.out‚0 disk.dm-0.out‚9341.212890625 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 1.2265625 disk.sda.out‚ 94.51171875$ proc.memory.availableMB‚477.8125) disk./.usagePercent‚74.47671752792243 disk.dm-0.in‚ 162.94140625 disk.dm-1.out‚ 94.51171875 disk.rbd3.out‚0 disk.rbd2.out‚ 54.8515625 disk.drbd1777.in‚0) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17 network.recv‚907288 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.26171875 disk.sda2.in‚0 disk.rbd3.in‚02é°Ôž±‚(:“ ¢¬æÔè®Õ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.301' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.260199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÿ´>²‚(:³ ¢¬æÔÊýØ disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1109.760051727295 disk.sda3.in‚ 0.01171875% disk.nvme0n1.out‚9345.40869140625 disk.dm-1.out‚ 95.00390625 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.26171875 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625) proc.memory.percent‚17.54920482635498" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0( proc.memory.availableMB‚ 477.76171875 disk.sda1.in‚0' disk.nvme0c0n1.out‚9345.40869140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 network.sent‚ 6.402437e+06 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 95.00390625 network.recv‚912747 disk.rbd2.out‚54.95361328125$ memory_percent‚97.08397388458252) disk./.usagePercent‚74.47692822212312 disk.sda3.out‚ 95.00390625 disk.nvme0n1.in‚ 162.94140625" disk.dm-0.out‚9345.40869140625 disk.rbd1.in‚0 cpu‚25.314836587476485 „”2³- _wandbruntime‚2441  _runtime‚2441B×þÇ´Á & lr-AdamW/pg2‚0.0001702041421078912 trainer/global_step‚3149 % _timestamp‚1.7884503498481567e+09 _runtime‚2441.402587522 _step‚254 & lr-AdamW/pg1‚0.0001702041421078912_d}ÌµÆ & lr-AdamW/pg2‚0.0001702041421078912 trainer/global_step‚3149 % _timestamp‚1.7884503498481567e+09 _runtime‚2441.402587522 _step‚254 & lr-AdamW/pg1‚0.0001702041421078912þ‘?Ë2¶- _wandbruntime‚2441  _runtime‚2441uJ·¦ ·×O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59279655 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198992797434368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45519554 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41581680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201751891295232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2618223291221 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184798119640064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184797445275136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41581240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59279655 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 F@DeviceStatsMonitor.on_train_bá ¹ÓÊatch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182169550482944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15026836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59259618 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59259618 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15027274 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚3149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45519554 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884503498489132e+09 _step‚255 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438¾ˆ%â'¸ÜO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59279655 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198992797434368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45519554 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41581680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201751891295232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2618223291221 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184798119640064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184797445275136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41581240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59279655 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182169550482944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15026836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59259618 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59259618 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15027274 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚3149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2618877728973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45519554 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884503498489132e+09 _step‚255 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ÿ­F2¹- _wandbruntime‚2442  _runtime‚2442$P[å&ºßM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184855292120064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26562572 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619039726185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2628714220032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182225921524736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59298135 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177465629517824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2629370595328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182225903521792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45533570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788450350515746e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649ù‘&»äM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184855292120064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚184854617741824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26562572 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104831705 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619039726185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2628714220032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182225921524736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59298135 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1822§.wsÐ25921524736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2759953981440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177465629517824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2629370595328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182225903521792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45533570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788450350515746e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180084652204649 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296€˜4¹™2¼- _wandbruntime‚2442  _runtime‚2442ÚU½Ï /loss/FBt3/detection_loss‚2.6567153334617615 epoch‚2 1loss/SRqqt3/detection_loss‚1.8384519390485905 .loss/FBt4/detection_loss‚2.925812712082496 trainer/global_step‚3149 0loss/FBt4/assignment_loss‚0.6323679639656873 /loss/FRt4/assignment_loss‚4.803656930015201 2loss/SRqqt1/assignment_loss‚2.0372936632104066 2loss/SRqqt3/assignment_loss‚1.6983199051270883 %loss/total_loss‚48.03667689177897 _runtime‚ 2442.07225508 _step‚257 /loss/FRt3/assignment_loss‚3.672088513983057 2loss/SRqqt4/assignment_loss‚2.3915453325166847 /loss/FRt3/detection_loss‚1.9475832892225144 1loss/SRqqt4/detection_loss‚2.6712865648847637 -loss/FBt2/detection_loss‚0.69138744528047 # _timestamp‚1.78845035051794e+09 0loss/FRt2/assignment_loss‚1.8666874028868594 /loss/FBt1/detection_loss‚2.0315183281898497 0loss/FBt3/assignment_loss‚0.7932957119139055 .loss/FRt1/detection_loss‚2.260768275778249 2loss/SRqqt2/assignment_loss‚1.1755955662698514 0loss/FBt1/assignment_loss‚0.7568696637948354 .loss/FRt2/detection_loss‚1.667476619704295 0loss/FBt2/assignment_loss‚0.1809814877189203 /loss/FRt4/detection_loss‚3.3002652611051286 1loss/SRqqt1/detection_loss‚1.7633864146714309 0loss/FRt1/assignment_loss‚3.1882133348932804 1loss/SRqqt2/detection_loss‚1.0851092320536413pU 6Z¾Ô 0loss/FRt1/assignment_loss‚3.1882133348932804 1loss/SRqqt2/detection_loss‚1.0851092320536413 /loss/FBt3/detection_loss‚2.6567153334617615 epoch‚2 1loss/SRqqt3/detection_loss‚1.8384519390485905 .loss/FBt4/detection_loss‚2.925812712082496 trainer/global_step‚3149 0loss/FBt4/assignment_loss‚0.6323679639656873 /loss/FRt4/assignment_loss‚4.803656930015201 2loss/SRqqt1/assignment_loss‚2.0372936632104066 2loss/SRqqt3/assignment_loss‚1.6983199051270883 %loss/total_loss‚48.03667689177897 _runtime‚ 2442.07225508 _step‚257 /loss/FRt3/assignment_loss‚3.672088513983057 2loss/SRqqt4/assignment_loss‚2.3915453325166847 /loss/FRt3/detection_loss‚1.9475832892225144 1loss/SRqqt4/detection_loss‚2.6712865648847637 -loss/FBt2/detection_loss‚0.69138744528047 # _timestamp‚1.78845035051794e+09 0loss/FRt2/assignment_loss‚1.8666874028868594 /loss/FBt1/detection_loss‚2.0315183281898497 0loss/FBt3/assignment_loss‚0.7932957119139055 .loss/FRt1/detection_loss‚2.260768275778249 2loss/SRqqt2/assignment_loss‚1.1755955662698514 0loss/FBt1/assignment_loss‚0.7568696637948354 .loss/FRt2/detection_loss‚1.667476619704295 0loss/FBt2/assignment_loss‚0.1809814877189203 /loss/FRt4/detection_loss‚3.3002652611051286 1loss/SRqqt1/detection_loss‚1.7633864146714309Þ$bb¿‚(:’ ±¬æÔ«°µÖ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.889999999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zsÓ >À‚(:³ ±¬æÔäêßÙ disk.sda3.out‚ 95.57421875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9442.93505859375% disk.nvme0n1.out‚9442.93505859375" disk.dm-0.out‚9442.93505859375$ memory_percent‚97.08399772644043 network.sent‚ 6.408059e+06 network.recv‚917050 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 162.94140625) proc.memory.percent‚17.54920482635498 disk.rbd1.out‚ 1.2265625 disk.rbd3.in‚0 disk.sda.out‚ 95.57421875 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0' proc.memory.availableMB‚ 477.7578125 cpu‚25.313111056241745 disk.dm-1.in‚ 0.01171875 disk.drbd1777.out‚0% disk./.usageGB‚1109.7600326538086 disk.sda1.in‚0# proc.memory.rssMB‚ 2875.26171875 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.out‚ 95.57421875 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚54.95361328125 disk.drbd1777.in‚0) disk./.usagePercent‚74.47692694208544ǠΓÁ‚(:ˆ À¬æÔõá¹× gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚148.842' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»®,(7‚(:¬ À¬æÔꣽØ disk.dm-1.out‚ 96.01953125 network.recv‚923467 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2875.26171875 disk.sda2.in‚0 disk.rbd1.out‚ 1.2265625 disk.rbd2.out‚56.60595703125 network.sent‚ 6.51688e+06) disk./.usagePercent‚74.47692694208544 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0 disk.sda.out‚ 96.01953125& disk.nvme0c0n1.out‚9443.5146484375 disk.sda1.in‚0$ disk.nvme0n1.out‚9443.5146484375! disk.dm-0.out‚9443.5146484375$ memory_percent‚97.08385467529297% disk./.usageGB‚1109.7600326538086 disk.sda3.out‚ 96.01953125 disk.rbd3.in‚0% proc.memory.availableMB‚ 477.78125 cpu‚25.31154990769856 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd0.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.54920482635498 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875µpGžÃ‚(:“ ϬæÔ»ÀÖÕ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.367' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.673399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Rcí8Ä‚(:­ ϬæÔÜç±Ø& disk.nvme0c0n1.out‚9540.9912109375 disk.sda2.in‚0 disk.sda3.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 96.59765625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 162.94140625 disk.sda.out‚ 96.59765625) disk./.usagePercent‚74.47692822212312$ memory_percent‚97.08280563354492& proc.memory.availableMB‚ 477.953125# proc.memory.rssMB‚ 2875.26171875 disk.sda1.in‚0 network.sent‚ 6.521992e+06$ disk./.usageGB‚1109.760051727295 disk.rbd2.out‚56.60595703125 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 network.recv‚927213 disk.rbd1.in‚0 disk.sda3.out‚ 96.59765625) proc.memory.percent‚17.54920482635498 cpu‚25.30988297587859$ disk.nvme0n1.out‚9540.9912109375 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625! disk.dm-0.out‚9540.9912109375‘hÓ2Å- _wandbruntime‚2476  _runtime‚2476{'BÉÆà ' lr-AdamW/pg1‚0.00016948932303590907 ' lr-AdamW/pg2‚0.00016948932303590907 trainer/global_step‚3199 % _timestamp‚1.7884503853631449e+09 _runtime‚2476.917520237 _step‚258ûÒ¡ÎÇÈ _runtime‚2476.917520237 _step‚258 ' lr-AdamW/pg1‚0.00016948932303590907 ' lr-AdamW/pg2‚0.00016948932303590907 trainer/global_step‚3199 % _timestamp‚1.7884503853631449e+09‚†e$­2È- _wandbruntime‚2476  _runtime‚2476ÍryèÜ'ÉÖO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106406022 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42217632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884503853638678e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184987202425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60185670 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202075358257664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204877728385536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106426061 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182814973506533 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26962804 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 _step‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 j÷%ÊÛO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180155914875904 _runtime‚ 2476.91847684 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184987220428288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2659058630629 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2669536960000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106406022 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42217632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 % _timestamp‚1.7884503853638678e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184987202425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60185670 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202075358257664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204877728385536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184987202425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106426061 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182814973506533 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2669536960000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26962804 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 _step‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60205707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46234368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204940223152640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182871365549377 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185043573467136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106458557 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46234368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187713929868288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187713255490048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185043555464192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2660529503553 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2477.584575931 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46234370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884503860297189e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60204124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106438492 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2669700025856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60224187 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202136985055232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42230278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180210836045824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60224187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803214157312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670356401152 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180210836045824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180210819006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60224187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803214157312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670356401152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46234368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204940223152640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182871365549377 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185043573467136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106458557 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46234368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187713929868288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187713255490048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185043555464192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46234370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2660529503553 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚2477.584575931 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46234370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884503860297189e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60204124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106438492 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2669700025856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60224187 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202136985055232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42230278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EÒ‚(:³ Þ¬æÔ¨¢ŠØ) disk./.usagePercent‚74.47692975816832 disk.sda.out‚ 97.23828125 disk.sda3.out‚ 97.23828125 disk.rbd2.in‚0% disk.nvme0n1.out‚9638.64990234375 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0% disk./.usageGB‚1109.7600746154785 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.dm-1.out‚ 97.23828125 disk.rbd3.in‚0 network.recv‚931226 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 56.7041015625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0# proc.memory.rssMB‚ 2875.26171875" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875( proc.memory.availableMB‚ 477.97265625) proc.memory.percent‚17.54920482635498 cpu‚25.308339845283843 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚9638.64990234375$ memory_percent‚97.08268642425537 network.sent‚ 6.528156e+06 disk.rbd0.out‚0" disk.dm-0.out‚9638.64990234375 disk.rbd1.out‚ 1.2265625ªcš“Ó‚(:ˆ í¬æÔ“¤ÙÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÎÐ?:Ô‚(:¯ í¬æÔÕú‹Ø" disk.dm-0.out‚9638.97412109375 disk.sda2.in‚0% disk./.usageGB‚1109.7601013183594 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0$ memory_percent‚97.08213806152344 disk.rbd1.out‚ 1.2265625" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.54920482635498 cpu‚25.306808268148522) disk./.usagePercent‚74.47693155022105 disk.dm-1.out‚ 97.63671875 disk.rbd3.in‚0 disk.sda.out‚ 97.63671875 disk.rbd0.in‚0 disk.rbd2.out‚ 56.7041015625 disk.dm-0.in‚ 162.94140625 disk.sda1.out‚0 network.recv‚937496 disk.rbd2.in‚0' disk.nvme0c0n1.out‚9638.97412109375% disk.nvme0n1.out‚9638.97412109375 network.sent‚ 6.636463e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 97.63671875$ proc.memory.availableMB‚478.0625# proc.memory.rssMB‚ 2875.26171875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 162.94140625ŽÎÚ´2Õ- _wandbruntime‚2512  _runtime‚2512.ÉÖà ' lr-AdamW/pg2‚0.00016875838493651097 trainer/global_step‚3249 % _timestamp‚1.7884504210644746e+09 _runtime‚2512.618870387 _step‚262 ' lr-AdamW/pg1‚0.00016875838493651097RøaÎ×È % _timestamp‚1.7884504210644746e+09 _runtime‚2512.618870387 _step‚262 ' lr-AdamW/pg1‚0.00016875838493651097 ' lr-AdamW/pg2‚0.00016875838493651097 trainer/global_step‚3249† :–Ê2Ø- _wandbruntime‚2512  _runtime‚2512øÝA¨Ù×O F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27371204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182901138443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190515395136512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46921152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205157895140864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2709866403840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285241)ùæR51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190514720771584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182901121403904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187804872370688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185601686851053 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2710522765824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 % _timestamp‚1.7884504210652342e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚263 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185601015373941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208003209784832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2700548407789 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 Ï×7â'ÚÜO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182901138443264 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190515395136512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46921152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205157895140864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2709866403840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190514720771584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208003391057920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182901121403904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187804872370688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185601686851053 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2710522765824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 % _timestamp‚1.7884504210652342e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61131759 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚263 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚3249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185601015373941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208003209784832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2700548407789 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27371204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414‡¨Fo’2Û- _wandbruntime‚2513  _runtime‚2513§U¶gç&ÜáM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 _runtime‚2513.290656081 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2701364842961 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2711342206976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚264 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2711342206976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884504217355254e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚3249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46935170 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46935168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344Õ–ú¯­ÝæM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2711342206976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884504217355254e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚3249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46935170 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46935168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 _runtime‚2513.290656081 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182956042573824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2701364842961 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.alloGŸô©?cated‚ 2711342206976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182956025534464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177065472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚264 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208240432024064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193373358519808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27779604 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 trainer/global_step‚3299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47621952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚2548.131891067 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27779606 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414¢tW‚â'êÜO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15708885 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185646344971264 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15708456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185646327931904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43488060 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47621952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚2548.131891067 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27779606 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43488491 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2751508571648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62057811 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193374032884736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2741383747197 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62037774 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2888231431168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47621954 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2740729309445 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208240432024064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193373358519808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27779604 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 trainer/global_step‚3299 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109692196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 L>DeviceStatsMonimBtor.on_train_batch_end/allocation.all.allocated‚ 109712261 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884504572447896e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _runtime‚2548.799698366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚268 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62056228 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211191337303040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193431205364736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2752328012800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188443449284193 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0÷˜¢Žì&íæM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884504572447896e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _runtime‚2548.799698366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚268 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62056228 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211191337303040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193431205364736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2752328012800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188443449284193 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15712943 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2741545744409 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47635968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15713386 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190678859348992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109692196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504ŒbŒj2î- _wandbruntime‚2548  _runtime‚2548ðI ÏYïÓ /loss/FRt4/assignment_loss‚5.529612609318325 2loss/SRqqt2/assignment_loss‚1.1925483160253083 .loss/FRt1/detection_loss‚2.190172894493393 epoch‚2 /loss/FRt1/assignment_loss‚2.925045994312867 0loss/FRt2/assignment_loss‚1.9191269442146899 0loss/FRt3/assignment_loss‚3.0279837082017145 1loss/SRqqt4/assignment_loss‚2.278776598699165 0loss/FBt1/assignment_loss‚0.2209635078907013 1loss/FBt3/assignment_loss‚0.43868697236757725 trainer/global_step‚3299 1loss/FBt2/assignment_loss‚0.34418357729437316 0loss/FBt4/assignment_loss‚0.3455861591942157 % _timestamp‚1.7884504572468657e+09 /loss/FRt2/detection_loss‚1.8040036063860445 .loss/FRt4/detection_loss‚3.286321134794326 /loss/FBt1/detection_loss‚3.8010629415512085 _step‚269 2loss/SRqqt1/assignment_loss‚1.6181458133671964 1loss/SRqqt1/detection_loss‚1.7329046428203583 .loss/FBt3/detection_loss‚2.531721064677605 _runtime‚2548.801161002 2loss/SRqqt3/assignment_loss‚2.0659206367201275 1loss/SRqqt3/detection_loss‚1.9000379774305556 .loss/FBt4/detection_loss‚3.190809515806345 %loss/total_loss‚48.96682529467571 .loss/FRt3/detection_loss‚2.066768873974962 1loss/SRqqt2/detection_loss‚1.3434560100947108 /loss/FBt2/detection_loss‚0.7498816517527419 1loss/SRqqt4/detection_loss‚2.4631041432871963yÈvá^ðØ 1loss/FBt3/assignment_loss‚0.43868697236757725 trainer/global_step‚3299 1loss/FBt2/assignment_loss‚0.34418357729437316 0loss/FBt4/assignment_loss‚0.3455861591942157 % _timestamp‚1.7884504572468657e+09 /loss/FRt2/detection_loss‚1.8040036063860445 .loss/FRt4/detection_loss‚3.286321134794326 /loss/FBt1/detection_loss‚3.8010629415512085 _step‚269 2loss/SRqqt1/assignment_loss‚1.6181458133671964 1loss/SRqqt1/detection_loss‚1.7329046428203583 .loss/FBt3/detection_loss‚2.531721064677605 _runtime‚2548.801161002 2loss/SRqqt3/assignment_loss‚2.0659206367201275 1loss/SRqqt3/detection_loss‚1.9000379774305556 .loss/FBt4/detection_loss‚3.190809515806345 %loss/total_loss‚48.96682529467571 .loss/FRt3/detection_loss‚2.066768873974962 1loss/SRqqt2/detection_loss‚1.3434560100947108 /loss/FBt2/detection_loss‚0.7498816517527419 1loss/SRqqt4/detection_loss‚2.4631041432871963 /loss/FRt4/assignment_loss‚5.529612609318325 2loss/SRqqt2/assignment_loss‚1.1925483160253083 .loss/FRt1/detection_loss‚2.190172894493393 epoch‚2 /loss/FRt1/assignment_loss‚2.925045994312867 0loss/FRt2/assignment_loss‚1.9191269442146899 0loss/FRt3/assignment_loss‚3.0279837082017145 1loss/SRqqt4/assignment_loss‚2.278776598699165 0loss/FBt1/assignment_loss‚0.2209635078907013qc¸’ñ‚(:‡ š­æÔëÇñÕ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o„x:ò‚(:¯ š­æÔï´›Ù% disk./.usageGB‚1109.7598571777344 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625) proc.memory.percent‚17.54920482635498% disk.nvme0n1.out‚9834.52685546875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 99.3046875 disk.rbd2.out‚ 58.5595703125# proc.memory.rssMB‚ 2875.26171875 network.sent‚ 6.756721e+06) disk./.usagePercent‚74.47691516573886 disk.sda3.out‚ 99.3046875" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.out‚ 99.3046875 cpu‚25.302201694175483 disk.sda1.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚9834.52685546875 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.rbd1.out‚ 1.2265625 disk.drbd1777.in‚0$ memory_percent‚97.08504676818848 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚9834.52685546875 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 477.5859375 disk.sda1.out‚0 network.recv‚951773£Èûžó‚(:“ ©­æÔÞߥ× gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.161' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.032200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆEc5ô‚(:ª ©­æÔòÒ…Ø disk.rbd0.in‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚9835.2158203125! disk.dm-0.out‚9835.2158203125 disk.dm-1.out‚ 99.796875 disk.sda.in‚ 0.01171875 disk.sda.out‚ 99.796875# proc.memory.rssMB‚ 2875.26171875) disk./.usagePercent‚74.47691516573886 disk.sda1.out‚0 cpu‚25.300765043600958" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 network.recv‚958532 disk.rbd2.out‚ 58.5595703125$ disk.nvme0n1.out‚9835.2158203125 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 99.796875( proc.memory.availableMB‚ 477.68359375 proc.cpu.threads‚17 network.sent‚ 6.865023e+06% disk./.usageGB‚1109.7598571777344 disk.rbd1.out‚ 1.2265625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚97.08445072174072) proc.memory.percent‚17.54920482635498÷\€ƒ“õ‚(:ˆ ¸­æÔ°³®Õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ÖBi7ö‚(:¬ ¸­æÔ¨âñØ network.sent‚ 6.871251e+06 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 100.265625 disk.rbd3.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚9932.6767578125 disk.rbd2.out‚58.66552734375! disk.dm-0.out‚9932.6767578125 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚9932.6767578125' proc.memory.availableMB‚ 477.4140625# proc.memory.rssMB‚ 2875.26171875 disk.rbd1.out‚ 1.2265625) disk./.usagePercent‚74.47691081361079 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0$ memory_percent‚97.08609580993652 disk.drbd1777.out‚0 network.recv‚963029 disk.sda.out‚ 100.265625 disk.sda1.out‚0 disk.dm-0.in‚ 162.94140625) proc.memory.percent‚17.54920482635498 cpu‚25.29930462798811 proc.cpu.threads‚17% disk./.usageGB‚1109.7597923278809 disk.rbd1.in‚0 disk.sda3.out‚ 100.265625 disk.rbd0.in‚0#Ÿâ2÷- _wandbruntime‚2583  _runtime‚2583ÂS)Éøà _step‚270 ' lr-AdamW/pg1‚0.00016724876428852079 ' lr-AdamW/pg2‚0.00016724876428852079 trainer/global_step‚3349 % _timestamp‚1.7884504924176424e+09 _runtime‚2583.972039924STñþÎùÈ ' lr-AdamW/pg1‚0.00016724876428852079 ' lr-AdamW/pg2‚0.00016724876428852079 trainer/global_step‚3349 % _timestamp‚1.7884504924176424e+09 _runtime‚2583.972039924 _step‚270Ž¡æÓœ2ú- _wandbruntime‚2583  _runtime‚2583£?i.P#ûÖO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15937488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48322752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚271 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48322754 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 =6DeviceStatsMonitor.on_train_batch_start/mar_ííŒx_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191173099108757 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Q6îrá'üÛO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788450492418396e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62963826 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 trainer/global_step‚3349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48322754 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191173099108757 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15937488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48322752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚271 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3349 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884504931048505e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62982280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2792657443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214316774499840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2584.659727049 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240Y’ATì&ÿæM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884504931048505e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62982280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2783035521777 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2792657443328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62982280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214316774499840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2584.659727049 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2793313818624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196289843112960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2782381083817 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚193496529294336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15942059 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196289168734720 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48336770 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3349 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974531307520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 % _timestamp‚1.7884505281212606e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚2619.675910587 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28596404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112933469 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430ÐÎÁâ'ŒÜO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112933469 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63909915 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196257810194944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112913430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63889878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974531307520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 % _timestamp‚1.7884505281212606e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49023554 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚2619.675910587 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28596404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2832823821312“Ÿ=’ð2- _wandbruntime‚2620  _runtime‚2620atL’ŽáM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112945900 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49037568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112945900 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 % _timestamp‚1.7884505287949288e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2833643249152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 _step‚276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214467156528128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112965965 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975241125888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199147806482944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2823216423225 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199148480861184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2833643249152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚2620.350026905 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296I»û ì&æM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28604574 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63908332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16170344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚3399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112965965 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975241125888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199147806482944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2823216423225 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975396348416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199148480861184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2833643249152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2823870861185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚2620.350026905 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112945900 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49037568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112945900 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 % _timestamp‚1.7884505287949288e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2833643249152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 _step‚276 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214467156528128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114560321 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193881964555264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392372 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120©‘½ÁžÜO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199075480140288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220505515993600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199075462137344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_b_‚VÈytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2873809627136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392372 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49724352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29004806 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114560321 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193881964555264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201949946129408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220567844966400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199131815176192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874629054976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _step‚280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199131833179136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220568032517632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199131815176192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ï´+Eë&¡åM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220568032517632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199131815176192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29012974 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217549693411328 _runtime‚ 2656.04048847 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2875285430272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193936851646464 % _timestamp‚1.7884505644855995e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193936868685824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49738370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220567844966400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199131815176192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29012972 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2874629054976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _step‚280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199131833179136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64854447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0˜¾÷ê2¢- _wandbruntime‚2656  _runtime‚2656ïQDµY£Ó 0loss/FRt4/assignment_loss‚4.7096627517180005 2loss/SRqqt3/assignment_loss‚1.5034486098145996 0loss/FRt2/assignment_loss‚1.5494400560855865 0loss/FRt3/assignment_loss‚3.6343559503555296 0loss/FBt3/assignment_loss‚0.7713439530796475 1loss/SRqqt3/detection_loss‚1.6898126687716555 -loss/FBt4/detection_loss‚2.57866176366806 2loss/SRqqt1/assignment_loss‚1.5907616277604744 0loss/FBt4/assignment_loss‚0.5613921713149163 .loss/FRt4/detection_loss‚3.362065699967471 1loss/SRqqt1/detection_loss‚1.7135102210773363 _step‚281 2loss/SRqqt2/assignment_loss‚1.0464826026472493 1loss/SRqqt4/assignment_loss‚2.149990846713384 0loss/FBt2/assignment_loss‚0.3093014196821808 /loss/FRt3/detection_loss‚2.3601974450051784 .loss/FBt1/detection_loss‚2.624602368899754 /loss/FBt2/detection_loss‚0.5678956339904329 epoch‚2 0loss/FBt1/assignment_loss‚0.3708010683102267 1loss/SRqqt4/detection_loss‚2.6952674609643443 /loss/FRt2/detection_loss‚1.8581416939313595 trainer/global_step‚3449 _runtime‚2656.041540873 $ _timestamp‚1.788450564487225e+09 0loss/FRt1/assignment_loss‚2.7868085343663287 .loss/FRt1/detection_loss‚2.362260813029801 /loss/FBt3/detection_loss‚2.6628790696461997 &loss/total_loss‚46.721516457921965 1loss/SRqqt2/detection_loss‚1.2624320271222487Ns¹^¤Ø /loss/FBt3/detection_loss‚2.6628790696461997 &loss/total_loss‚46.721516457921965 1loss/SRqqt2/detection_loss‚1.2624320271222487 0loss/FRt4/assignment_loss‚4.7096627517180005 2loss/SRqqt3/assignment_loss‚1.5034486098145996 0loss/FRt2/assignment_loss‚1.5494400560855865 0loss/FRt3/assignment_loss‚3.6343559503555296 0loss/FBt3/assignment_loss‚0.7713439530796475 1loss/SRqqt3/detection_loss‚1.6898126687716555 -loss/FBt4/detection_loss‚2.57866176366806 2loss/SRqqt1/assignment_loss‚1.5907616277604744 0loss/FBt4/assignment_loss‚0.5613921713149163 .loss/FRt4/detection_loss‚3.362065699967471 1loss/SRqqt1/detection_loss‚1.7135102210773363 _step‚281 2loss/SRqqt2/assignment_loss‚1.0464826026472493 1loss/SRqqt4/assignment_loss‚2.149990846713384 0loss/FBt2/assignment_loss‚0.3093014196821808 /loss/FRt3/detection_loss‚2.3601974450051784 .loss/FBt1/detection_loss‚2.624602368899754 /loss/FBt2/detection_loss‚0.5678956339904329 epoch‚2 0loss/FBt1/assignment_loss‚0.3708010683102267 1loss/SRqqt4/detection_loss‚2.6952674609643443 /loss/FRt2/detection_loss‚1.8581416939313595 trainer/global_step‚3449 _runtime‚2656.041540873 $ _timestamp‚1.788450564487225e+09 0loss/FRt1/assignment_loss‚2.7868085343663287 .loss/FRt1/detection_loss‚2.362260813029801™}Ìã ’¥‚(:‡ ’®æÔ™™ÝÕ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚98.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ÅW=¦‚(:² ’®æÔÿ»”Ù network.sent‚ 7.111056e+06" disk.dm-0.out‚10227.4892578125 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 477.28125# proc.memory.rssMB‚ 2875.26171875' disk.nvme0c0n1.out‚10227.4892578125 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 103.11328125% disk.nvme0n1.out‚10227.4892578125 disk.drbd1777.out‚0 disk.dm-1.out‚ 103.11328125 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625$ memory_percent‚97.08690643310547) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17% disk./.usageGB‚1109.7599983215332 disk.sda.out‚ 103.11328125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚992370 disk.rbd2.out‚ 60.587890625 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 1.2265625 disk.nvme0n1.in‚ 162.94140625 cpu‚25.290882977522543) disk./.usagePercent‚74.47692463801764 disk.sda2.in‚0 disk.rbd1.in‚0«È+Ú7§‚(:ˆ ¡®æÔ°÷ÏÕ gpu.0.gpu‚43.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5086 gpu.0.smClock‚1410 gpu.xA<\0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¬­ç=¨‚(:² ¡®æÔåàØ disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd2.out‚ 60.587890625 disk.nvme0n1.in‚ 162.94140625& proc.memory.availableMB‚ 476.859375# proc.memory.rssMB‚ 2875.26171875 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda.in‚ 0.01171875 disk.sda1.in‚0) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17% disk./.usageGB‚1109.7602615356445# disk.dm-0.out‚10325.53662109375 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚10325.53662109375 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚10325.53662109375 network.recv‚998237 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 162.94140625 network.sent‚ 7.219149e+06( disk./.usagePercent‚74.4769423025375 disk.dm-1.out‚ 103.6484375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 103.6484375 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚97.08948135375977 cpu‚25.289567887178215 disk.sda2.in‚0 disk.sda.out‚ 103.6484375%lå2©- _wandbruntime‚2691  _runtime‚2691Œ;Ȫ $ _timestamp‚1.788450599685015e+09 _runtime‚2691.239394862 _step‚282 ' lr-AdamW/pg1‚0.00016486735622671435 ' lr-AdamW/pg2‚0.00016486735622671435 trainer/global_step‚3499@ QËÍ«Ç ' lr-AdamW/pg1‚0.00016486735622671435 ' lr-AdamW/pg2‚0.00016486735622671435 trainer/global_step‚3499 $ _timestamp‚1.788450599685015e+09 _runtime‚2691.239394862 _step‚282š¹õ¼:2¬- _wandbruntime‚2691  _runtime‚2691Ù7­$Ý'­×O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚2691.240742631 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50425152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199531224710981 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65741982 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚283 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201893132082688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50425154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201893114079744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3060489982976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2904070667077 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884505996859875e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3060653607424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2914795432960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65741982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220570555616768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199531896188093 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196627154043904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46033415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 trainer/global_step‚3499 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688Û´`Ãâ'®ÜO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3060653607424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2914795432960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65741982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220570555616768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199531896188093 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196627154043904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46033415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 trainer/global_step‚3499 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚2691.240742631 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50425152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2914795432960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199531224710981 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50425154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65741982 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚283 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201893132082688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50425154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201893114079744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29413204 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3060489982976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2904070667077 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884505996859875e+09›Í‚%p2¯- _wandbruntime‚2691  _runtime‚2691檯ç&°áM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16624657 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204865756357632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3061517095936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201949485121536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65780499 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204865081979392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚3499 _step‚284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚204865756357632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3061353484800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚2691.913484907 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223693559839232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884506003583524e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65760436 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029‘AI±æM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204865756357632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50439168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3061517095936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199587616753825 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201949485121536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220632230294528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65780499 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚204865081979392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196682075213824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚3499 _step‚284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚204865756357632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3061353484800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚2691.913484907 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223693559839232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884506003583524e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65760436 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16624657 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432œ2ªÖ–2²- _wandbruntime‚2691  _runtime‚2691‡}lHZ³Ô 2loss/SRqqt2/assignment_loss‚0.9258691941937193 1loss/SRqqt2/detection_loss‚1.0340185232269459 _runtime‚2691.914819071 % _timestamp‚1.7884506003605216e+09 2loss/SRqqt1/assignment_loss‚1.5845527665651575 0loss/FBt3/assignment_loss‚0.3341985152411932 1loss/SRqqt1/detection_loss‚1.8185287659265557 .loss/FBt3/detection_loss‚3.288134843111038 /loss/FRt4/detection_loss‚3.1093899541431003 /loss/FBt1/detection_loss‚1.9849456275502841 0loss/FRt1/assignment_loss‚3.5907387844351835 .loss/FRt1/detection_loss‚2.440628543149593 trainer/global_step‚3499 2loss/SRqqt4/assignment_loss‚2.1877674693515634 epoch‚2 0loss/FRt2/assignment_loss‚1.6375947147607803 /loss/FRt4/assignment_loss‚4.922036647796631 0loss/SRqqt4/detection_loss‚2.575386835371747 /loss/FBt2/detection_loss‚0.7129971902398631 2loss/SRqqt3/assignment_loss‚1.9044043936301023 0loss/FBt4/assignment_loss‚0.4344678707207841 /loss/FRt2/detection_loss‚1.7041495781964981 .loss/FRt3/detection_loss‚1.941555498311153 1loss/SRqqt3/detection_loss‚1.8149532269065578 /loss/FBt4/detection_loss‚3.8997594118118286 _step‚285 /loss/FRt3/assignment_loss‚3.510608909221796 0loss/FBt1/assignment_loss‚0.6738770613083034 1loss/FBt2/assignment_loss‚0.26238335101975585 %loss/total_loss‚48.29294767619013›´`_´Ù 0loss/FRt2/assignment_loss‚1.6375947147607803 /loss/FRt4/assignment_loss‚4.922036647796631 0loss/SRqqt4/detection_loss‚2.575386835371747 /loss/FBt2/detection_loss‚0.7129971902398631 2loss/SRqqt3/assignment_loss‚1.9044043936301023 0loss/FBt4/assignment_loss‚0.4344678707207841 /loss/FRt2/detection_loss‚1.7041495781964981 .loss/FRt3/detection_loss‚1.941555498311153 1loss/SRqqt3/detection_loss‚1.8149532269065578 /loss/FBt4/detection_loss‚3.8997594118118286 _step‚285 /loss/FRt3/assignment_loss‚3.510608909221796 0loss/FBt1/assignment_loss‚0.6738770613083034 1loss/FBt2/assignment_loss‚0.26238335101975585 %loss/total_loss‚48.29294767619013 2loss/SRqqt2/assignment_loss‚0.9258691941937193 1loss/SRqqt2/detection_loss‚1.0340185232269459 _runtime‚2691.914819071 % _timestamp‚1.7884506003605216e+09 2loss/SRqqt1/assignment_loss‚1.5845527665651575 0loss/FBt3/assignment_loss‚0.3341985152411932 1loss/SRqqt1/detection_loss‚1.8185287659265557 .loss/FBt3/detection_loss‚3.288134843111038 /loss/FRt4/detection_loss‚3.1093899541431003 /loss/FBt1/detection_loss‚1.9849456275502841 0loss/FRt1/assignment_loss‚3.5907387844351835 .loss/FRt1/detection_loss‚2.440628543149593 trainer/global_step‚3499 2loss/SRqqt4/assignment_loss‚2.1877674693515634 epoch‚2õ|9“µ‚(:ˆ °®æÔ‡ÃÐÖ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.653' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìö>G¶‚(:¼ °®æÔòÌØ disk.sda2.in‚0 cpu‚25.288161953690672 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 7.225377e+06% disk./.usageGB‚1109.7602424621582$ memory_percent‚97.08681106567383( disk.nvme0c0n1.out‚10325.87255859375 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2875.26171875) proc.memory.percent‚17.54920482635498 disk.sda3.out‚ 104.06640625) disk./.usagePercent‚74.47694102249983" disk.nvme0c0n1.in‚ 162.94140625& proc.memory.availableMB‚ 477.296875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd1.out‚ 1.2265625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 104.06640625& disk.nvme0n1.out‚10325.87255859375 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 104.06640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda1.out‚0 network.recv‚ 1.002758e+06 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚ 62.240234375# disk.dm-0.out‚10325.87255859375¸]2ë’·‚(:‡ ¿®æÔÀ¤ØÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¹-D¸‚(:¹ ¿®æÔÉ©¸Ø network.sent‚ 7.333136e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0) disk./.usagePercent‚74.47701347263201 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625' proc.memory.availableMB‚ 477.0390625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 104.75390625 disk.rbd1.out‚ 1.2265625 disk.sda.in‚ 0.01171875 disk.sda1.out‚0' disk.nvme0c0n1.out‚10423.3994140625 cpu‚25.286850246573167 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0) proc.memory.percent‚17.54920482635498 disk.sda3.out‚ 104.75390625 network.recv‚ 1.008557e+06 disk.sda.out‚ 104.75390625 disk.drbd1777.out‚0 disk.rbd3.in‚0# memory_percent‚97.0883846282959% disk./.usageGB‚1109.7613220214844 disk.rbd2.out‚ 62.255859375% disk.nvme0n1.out‚10423.3994140625 disk.dm-0.in‚ 162.94140625" disk.dm-0.out‚10423.3994140625 disk.sda2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2875.26171875 disk.rbd1.in‚0Ô»2¹- _wandbruntime‚2727  _runtime‚2727„޸ɺà ' lr-AdamW/pg1‚0.00016404301812788884 ' lr-AdamW/pg2‚0.00016404301812788884 trainer/global_step‚3549 % _timestamp‚1.7884506355890896e+09 _runtime‚2727.143474117 _step‚286>è²9Î»È trainer/global_step‚3549 % _timestamp‚1.7884506355890896e+09 _runtime‚2727.143474117 _step‚286 ' lr-AdamW/pg1‚0.00016404301812788884 ' lr-AdamW/pg2‚0.00016404301812788884žM±d2¼- _wandbruntime‚2727  _runtime‚2727F˜ï Ý'½×O PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66668034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2955781238784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46669646 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202317938055501 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51125952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944906006485 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29821604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66668034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204710784025088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2956437600768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 trainer/global_step‚3549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207666547260928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204710784025088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2956437600768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 trainer/global_step‚3549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51125952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944906006485 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29821604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66668034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226756863156736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945560444237 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202373658621233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199427264702464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚3549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204767119060992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51139970 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517uaê&ÁäM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956600666624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51139970 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226819365154304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51139970 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29829774 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202373658621233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199427264702464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚3549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66686488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204767119060992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223714767177728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51139968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Š×¹-2Â- _wandbruntime‚2727  _runtime‚2727ÃW­g]Ã× 1loss/SRqqt3/detection_loss‚1.6933660990697272 /loss/FBt2/detection_loss‚0.6784465654850084 &loss/total_loss‚47.096536459634635 _step‚289 /loss/FRt4/assignment_loss‚4.329202651977539 1loss/FBt3/assignment_loss‚0.36346234782831743 /loss/FRt2/detection_loss‚1.5667464671766056 1loss/SRqqt2/detection_loss‚1.0109068904268115 .loss/FBt1/detection_loss‚2.659091079235077 /loss/FBt3/detection_loss‚2.7907740101218224 2loss/SRqqt1/assignment_loss‚1.6011370953093185 2loss/SRqqt4/assignment_loss‚2.3818912375718355 epoch‚2 trainer/global_step‚3549 2loss/SRqqt2/assignment_loss‚0.8673980957583377 2loss/SRqqt3/assignment_loss‚1.4833794933486175 _runtime‚2727.817333773 0loss/FRt1/assignment_loss‚3.8160916052081366 0loss/SRqqt4/detection_loss‚2.584076310507953 0loss/FRt2/assignment_loss‚1.8571827937574947 0loss/FRt3/assignment_loss‚3.3247412776947023 /loss/FRt3/detection_loss‚2.2360593938827513 1loss/FBt4/assignment_loss‚0.44436974972466603 .loss/FRt1/detection_loss‚2.634379355744882 0loss/FBt1/assignment_loss‚0.8935740336775779 1loss/SRqqt1/detection_loss‚1.8774960719524545 .loss/FBt4/detection_loss‚2.883726119995117 % _timestamp‚1.7884506362627285e+09 1loss/FBt2/assignment_loss‚0.29884624044973374 .loss/FRt4/detection_loss‚2.820191473730149D³›4bÄÜ 1loss/FBt2/assignment_loss‚0.29884624044973374 .loss/FRt4/detection_loss‚2.820191473730149 1loss/SRqqt3/detection_loss‚1.6933660990697272 /loss/FBt2/detection_loss‚0.6784465654850084 &loss/total_loss‚47.096536459634635 _step‚289 /loss/FRt4/assignment_loss‚4.329202651977539 1loss/FBt3/assignment_loss‚0.36346234782831743 /loss/FRt2/detection_loss‚1.5667464671766056 1loss/SRqqt2/detection_loss‚1.0109068904268115 .loss/FBt1/detection_loss‚2.659091079235077 /loss/FBt3/detection_loss‚2.7907740101218224 2loss/SRqqt1/assignment_loss‚1.6011370953093185 2loss/SRqqt4/assignment_loss‚2.3818912375718355 epoch‚2 trainer/global_step‚3549 2loss/SRqqt2/assignment_loss‚0.8673980957583377 2loss/SRqqt3/assignment_loss‚1.4833794933486175 _runtime‚2727.817333773 0loss/FRt1/assignment_loss‚3.8160916052081366 0loss/SRqqt4/detection_loss‚2.584076310507953 0loss/FRt2/assignment_loss‚1.8571827937574947 0loss/FRt3/assignment_loss‚3.3247412776947023 /loss/FRt3/detection_loss‚2.2360593938827513 1loss/FBt4/assignment_loss‚0.44436974972466603 .loss/FRt1/detection_loss‚2.634379355744882 0loss/FBt1/assignment_loss‚0.8935740336775779 1loss/SRqqt1/detection_loss‚1.8774960719524545 .loss/FBt4/detection_loss‚2.883726119995117 % _timestamp‚1.7884506362627285e+09¡§AÅÂÅ‚(:… ήæÔÆÖÌÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§­TCÆ‚(:¸ ήæÔ¾ãöØ cpu‚25.285551827074304% disk./.usageGB‚1109.7613220214844 disk.rbd1.out‚ 1.2265625% disk.nvme0n1.out‚10423.7158203125 network.sent‚ 7.338464e+06 disk.sda.out‚ 105.2265625 disk.rbd2.in‚0' disk.nvme0c0n1.out‚10423.7158203125" disk.dm-0.out‚10423.7158203125 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.out‚ 105.2265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625$ memory_percent‚97.09007740020752 network.recv‚ 1.011832e+06 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2875.26171875 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0) disk./.usagePercent‚74.47701347263201 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625) proc.memory.percent‚17.54920482635498 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 476.76171875 disk.rbd2.out‚ 62.380859375 disk.drbd1777.in‚0 disk.dm-1.out‚ 105.2265625ƒÓ‡“Ç‚(:ˆ Ý®æÔžããÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÓUá7È‚(:¬ Ý®æÔóäÆØ disk.rbd1.out‚ 1.2265625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.54920482635498$ disk./.usageGB‚1109.761302947998 disk.rbd0.out‚0 cpu‚25.284283467079412 proc.cpu.threads‚17! disk.dm-0.out‚10521.447265625 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0$ disk.nvme0n1.out‚10521.447265625 disk.sda.in‚ 0.01171875 proc.memory.availableMB‚477 disk.sda2.in‚0 network.sent‚ 7.446739e+06) disk./.usagePercent‚74.47701219259434 disk.sda.out‚ 105.94921875 disk.rbd2.in‚0 disk.rbd2.out‚ 62.396484375" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚10521.447265625 disk.sda3.out‚ 105.94921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.out‚ 105.94921875" memory_percent‚97.088623046875 network.recv‚ 1.018223e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 162.94140625# proc.memory.rssMB‚ 2875.261718759òˆA“É‚(:ˆ ì®æÔÃêÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦d?Ê‚(:´ ì®æÔ£äéØ$ disk./.usageGB‚1109.761302947998" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚10521.78759765625# disk.dm-0.out‚10521.78759765625 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 106.328125# proc.memory.rssMB‚ 2875.26171875 disk.sda2.in‚0$ memory_percent‚97.09007740020752 cpu‚25.28310659987228 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.rbd2.out‚62.48291015625 disk.dm-1.out‚ 106.328125 disk.rbd1.in‚0& disk.nvme0n1.out‚10521.78759765625 disk.dm-0.in‚ 162.94140625 disk.rbd1.out‚1.3125( proc.memory.availableMB‚ 476.76171875) proc.memory.percent‚17.54920482635498 network.sent‚ 7.453114e+06 disk.rbd2.in‚0 disk.sda3.out‚ 106.328125 disk.nvme0n1.in‚ 162.94140625 disk.sda3.in‚ 0.01171875 network.recv‚ 1.02264e+06) disk./.usagePercent‚74.47701219259434 disk.drbd1777.out‚0 proc.cpu.threads‚17x{<2Ë- _wandbruntime‚2763  _runtime‚2763ŸÌ:ÈÌ _runtime‚ 2763.00744749 _step‚290 ' lr-AdamW/pg1‚0.00016320369600342704 ' lr-AdamW/pg2‚0.00016320369600342704 trainer/global_step‚3599 % _timestamp‚1.7884506714530594e+09‘ÕµÙÍÍÇ ' lr-AdamW/pg1‚0.00016320369600342704 ' lr-AdamW/pg2‚0.00016320369600342704 trainer/global_step‚3599 % _timestamp‚1.7884506714530594e+09 _runtime‚ 2763.00744749 _step‚290¢äâ¸ã2Î- _wandbruntime‚2763  _runtime‚2763ã¤Ð“ÎÏÕO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2986395783645 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3599 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47306072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205103308445797 PCDeviceSt‰„ý "atsMonitor.on_train_batch_start/active.large_pool.allocated‚51826754 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51826752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226735653323264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207528435967488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3146748990464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207528417964544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119440877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67614123 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210525185009152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67594086 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2996767044608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207528417964544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 2763.00845548 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202117567099904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67594086 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207528435967488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788450671453762e+09 _step‚291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210525859374080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30230004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3146581171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2996767044608®½Qà'ÐÚO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207528435967488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3146748990464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207528417964544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119440877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67614123 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚210525185009152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67594086 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2996767044608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207528417964544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 2763.00845548 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202117567099904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67594086 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207528435967488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788450671453762e+09 _step‚291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210525859374080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30230004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3146581171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2996767044608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2986395783645 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3599 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47306072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205103308445797 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51826754 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51826752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226735653323264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0£|’ŠÀ2Ñ- _wandbruntime‚2763  _runtime‚2763í$Cæ&ÒàM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119453308 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚ 2763.68051868 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119453308 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2998242847744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226797304060928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884506721254592e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47319180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207584771003392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226797304060928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884506721254592e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51840770 OBDeviceStatsMonitor.on_train_batch_end/inactive ¯þ/_split.all.allocated‚47319180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚210583031854080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67632603 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207584771003392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2997586472448 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210582357475840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119453308 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226797280120832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚ 2763.68051868 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119453308 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2998242847744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚213948760022528 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158158848 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788450739055239e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3038278388736 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚208411387736681 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3027093726825j>6’ÎáÂT OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚208411387736681 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3027093726825 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚68489106 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚210910992466432 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚213948760022528 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158158848 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788450739055239e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3038278388736 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0¦s¼ðû2â- _wandbruntime‚2831  _runtime‚2831¬^+ÅN)ãÈR NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230355636712960 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 134218752 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚213985955177984 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚179 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884507400102727e+09 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚210947822495232 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815168 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3189193675264 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚205419587769344 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚52759502 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121239932 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 _runtime‚2831.564975245 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚210947804492288 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17424425 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699021 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚233544830388224 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914388 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 121239932 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚213986466010624 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚205419570729984 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚213985955177984 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585536 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230355660653056 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3626 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚52759502 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829696 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚68480432 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832640 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 _runtime‚2831.564975245 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚210947804492288 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17424425 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699021 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚233544988547072 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚233544830388224 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914388 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 121239932 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚213986466010624 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚205419570729984 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚213985955177984 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585536 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230355660653056 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3626 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚52759502 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3038643515392 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829696 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177§êáß‚žå‚(:“ ·¯æÔòé½Õ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚165.401' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.080200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁŸY*Hæ‚(:½ ·¯æÔàãø× network.sent‚ 7.584152e+06) disk./.usagePercent‚74.47701372863955 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17% disk./.usageGB‚1109.7613258361816% disk.nvme0n1.out‚10815.7900390625" disk.dm-0.out‚10815.7900390625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.54920482635498 cpu‚25.36973518718896 network.recv‚ 1.044572e+06" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.47021484375 disk.sda.out‚ 109.14453125 disk.rbd2.out‚ 64.2490234375 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.out‚ 109.14453125 disk.sda1.in‚0 disk.dm-0.in‚ 162.94140625 disk.rbd3.in‚0 disk.rbd0.out‚0$ memory_percent‚97.08852767944336 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 109.14453125 disk.rbd0.in‚0& proc.memory.availableMB‚ 477.015625# proc.memory.rssMB‚ 2875.26171875 disk.rbd2.in‚0' disk.nvme0c0n1.out‚10815.7900390625àY2ç- _wandbruntime‚2844  _runtime‚2844oS©>cèݧ 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010071806609630585 *jet/accuracy_1_of_3‚0.8612508773803711 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13974864780902863 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5753968358039856 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0636984184384346 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.982417643070221 7"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.53800368309021 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950245022773743 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11847139149904251 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 jet/accuracy_5_of_6‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5281380414962769 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16896820068359375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9754893183708191 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6379615068435669 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42621108889579773 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03355390951037407 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1927850842475891 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4751461446285248 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9698212146759033 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4662984311580658 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5165543556213379 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34689924120903015 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12011493742465973 +jet/accuracy_2_of_2‚0.10401120036840439 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5276223421096802 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5115247964859009 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2480158507823944 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 trainer/global_step‚3626 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9710692167282104 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1172318086028099 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2915724813938141 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666865348816 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444614648819 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01655174233019352 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3447916507720947 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301889076828957 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5128692388534546 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397¹µe÷|395521402359 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 jet/accuracy_5_of_5‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9486879706382751 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40223607420921326 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908414483070374 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0387430265545845 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5268757939338684 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1328578144311905 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9133355021476746 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 particle/accuracy_3_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1969268023967743 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9697612524032593 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18502362072467804 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48851969838142395 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14664727449417114 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6710192561149597 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9960792064666748 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9664682745933533 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5274808406829834 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5249097347259521 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1949203759431839 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5591055750846863 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.49226194620132446 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚296 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5153460502624512 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19290123879909515 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5265889167785645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28635239601135254 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5388699769973755 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39489486813545227 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49346598982810974 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44581785798072815 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878532290458679 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001727357623167336 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 ,particle/sensitivity‚0.28870731592178345 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44782140851020813 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 % _timestamp‚1.7884507526833377e+09 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1564098447561264 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.510161817073822 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212962746620178 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13829094171524048 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4758284389972687 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1303742229938507 particle/accuracy_4_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3510603904724121 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23005984723567963 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5260342359542847 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44172120094299316 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_7‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22316019237041473 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862750768661499 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15918384492397308 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05440652742981911 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5053420662879944 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5900346040725708 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22625884413719177 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9497076272964478 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2274646759033203 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004032257944345474 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1440843790769577 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11447038501501083 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4268483817577362 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13523191213607788 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9651361107826233 )jet/accuracy_2_of_5‚0.477150559425354 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5294561982154846 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5292256474494934 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0047744340263307095 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12574811279773712 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4760057330131531 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5892857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 _runtime‚2844.240141897 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9723798632621765 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10233917832374573 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5122582316398621 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9856411218643188 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4426285922527313 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4695690870285034 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06633489578962326 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936010241508484 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.613135576248169 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30169960856437683 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5641239881515503 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 particle/accuracy_2_of_7‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17775145173072815 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1696382611989975 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.013071895577013493 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24511495232582092 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1757090538740158 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9559160470962524 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9832422137260437 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5323257446289062 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8225842118263245 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4613984227180481 +jet/accuracy_3_of_4‚0.05452341586351395 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.49618053436279297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19639639556407928 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4099326431751251 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36246493458747864 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13390009105205536 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125295877456665 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013025950640439987 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3441358208656311 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3579774498939514 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3597222566604614 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5339946746826172 *jet/accuracy_1_of_2‚0.7506785988807678 (particle/f_score‚0.37806984782218933 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2578597664833069 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2802601754665375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09152241051197052 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01344396360218525 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027348952367901802 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5063980221748352 particle/accuracy_5_of_5‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_3_of_5‚0.1187916100025177 particle/accuracy_3_of_3‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8517277836799622 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011632319539785385 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03766420856118202 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_5‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930998086929321 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5018466711044312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3714924156665802 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5396987199783325 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0872984305024147 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13974864780902863 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9550691246986389 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08641975373029709 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18518893420696259 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17683784663677216 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890170097351074 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07858634740114212 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333230018616 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9759887456893921 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37061402201652527 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.4356183111667633 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09690242260694504 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0015495087718591094 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5632306933403015 )loss/val_total_loss‚48.47394654578681 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.981256365776062 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3258928656578064 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6315789222717285 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3274622857570648 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.365011990070343 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.520190954208374 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07932719588279724 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27595311403274536 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4882332682609558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12141910940408707 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.004567852709442377 particle/accuracy_1_of_2‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13174563646316528 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03351505100727081 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4871254861354828 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5066152215003967 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620977401733 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691343307495 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2805555760860443 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24570278823375702 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885656833648682 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13634702563285828 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22796140611171722 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03244047611951828 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_1_of_1‚0.48915010690689087 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_4_of_7‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45241934061050415 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4969135820865631 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903508186340332 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1588965356349945 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0023767082020640373 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5212020874023438 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,jet/accuracy_3_of_3‚0.018760526552796364 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6204844117164612 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12563642859458923 *jet/accuracy_2_of_3‚0.2529900074005127 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48770609498023987 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5268757939338684 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 epoch‚2 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5838197469711304 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31900954246520996 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5433389544487 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3476875126361847 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841970801353455 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18484848737716675 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12061534821987152 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5352879166603088£.îqé⧠@&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13829094171524048 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4758284389972687 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1303742229938507 particle/accuracy_4_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009g+ù920635493472219 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3510603904724121 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23005984723567963 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5260342359542847 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44172120094299316 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_7‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22316019237041473 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862750768661499 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15918384492397308 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05440652742981911 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5053420662879944 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5900346040725708 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22625884413719177 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9497076272964478 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2274646759033203 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004032257944345474 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1440843790769577 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11447038501501083 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4268483817577362 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13523191213607788 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9651361107826233 )jet/accuracy_2_of_5‚0.477150559425354 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5294561982154846 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5292256474494934 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0047744340263307095 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12574811279773712 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4760057330131531 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5892857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 _runtime‚2844.240141897 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9723798632621765 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10233917832374573 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5122582316398621 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9856411218643188 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4426285922527313 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4695690870285034 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06633489578962326 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936010241508484 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.613135576248169 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30169960856437683 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5641239881515503 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 particle/accuracy_2_of_7‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17775145173072815 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1696382611989975 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.013071895577013493 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24511495232582092 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1757090538740158 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9559160470962524 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9832422137260437 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5323257446289062 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8225842118263245 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4613984227180481 +jet/accuracy_3_of_4‚0.05452341586351395 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.49618053436279297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19639639556407928 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4099326431751251 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36246493458747864 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13390009105205536 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125295877456665 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013025950640439987 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3441358208656311 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3579774498939514 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3597222566604614 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5339946746826172 *jet/accuracy_1_of_2‚0.7506785988807678 (particle/f_score‚0.37806984782218933 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2578597664833069 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2802601754665375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09152241051197052 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01344396360218525 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027348952367901802 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5063980221748352 particle/accuracy_5_of_5‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_3_of_5‚0.1187916100025177 particle/accuracy_3_of_3‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8517277836799622 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011632319539785385 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03766420856118202 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_5‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930998086929321 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5018466711044312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3714924156665802 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5396987199783325 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0872984305024147 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13974864780902863 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9550691246986389 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08641975373029709 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18518893420696259 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17683784663677216 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890170097351074 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07858634740114212 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333230018616 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9759887456893921 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37061402201652527 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.4356183111667633 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09690242260694504 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0015495087718591094 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5632306933403015 )loss/val_total_loss‚48.47394654578681 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.981256365776062 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3258928656578064 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6315789222717285 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3274622857570648 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.365011990070343 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.520190954208374 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07932719588279724 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 jet/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27595311403274536 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4882332682609558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12141910940408707 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.004567852709442377 particle/accuracy_1_of_2‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13174563646316528 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03351505100727081 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4871254861354828 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5066152215003967 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620977401733 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691343307495 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2805555760860443 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24570278823375702 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885656833648682 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13634702563285828 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22796140611171722 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03244047611951828 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_1_of_1‚0.48915010690689087 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_4_of_7‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.45241934061050415 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4969135820865631 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903508186340332 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1588965356349945 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0023767082020640373 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5212020874023438 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,jet/accuracy_3_of_3‚0.018760526552796364 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6204844117164612 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12563642859458923 *jet/accuracy_2_of_3‚0.2529900074005127 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48770609498023987 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5268757939338684 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 epoch‚2 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5838197469711304 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31900954246520996 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5433389544487 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3476875126361847 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841970801353455 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18484848737716675 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12061534821987152 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5352879166603088 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010071806609630585 *jet/accuracy_1_of_3‚0.8612508773803711 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13974864780902863 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5753968358039856 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0636984184384346 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.982417643070221 7"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.53800368309021 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950245022773743 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11847139149904251 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 jet/accuracy_5_of_6‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5281380414962769 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16896820068359375 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9754893183708191 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6379615068435669 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42621108889579773 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03355390951037407 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1927850842475891 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4751461446285248 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9698212146759033 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4662984311580658 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5165543556213379 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34689924120903015 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12011493742465973 +jet/accuracy_2_of_2‚0.10401120036840439 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5276223421096802 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5115247964859009 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2480158507823944 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 trainer/global_step‚3626 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9710692167282104 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1172318086028099 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2915724813938141 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666865348816 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1644444614648819 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01655174233019352 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3447916507720947 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301889076828957 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5128692388534546 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397395521402359 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 jet/accuracy_5_of_5‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9486879706382751 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40223607420921326 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908414483070374 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0387430265545845 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5268757939338684 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1328578144311905 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9133355021476746 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.07692307978868484 particle/accuracy_3_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1969268023967743 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9697612524032593 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18502362072467804 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48851969838142395 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14664727449417114 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6710192561149597 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 PuritBÑÝy/1FRt*SRqqt1FBt/FBt_purity‚0.9960792064666748 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9664682745933533 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5274808406829834 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5249097347259521 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1949203759431839 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5591055750846863 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.49226194620132446 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚296 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5153460502624512 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19290123879909515 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5265889167785645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28635239601135254 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5388699769973755 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39489486813545227 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49346598982810974 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44581785798072815 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878532290458679 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.001727357623167336 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 ,particle/sensitivity‚0.28870731592178345 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.44782140851020813 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 % _timestamp‚1.7884507526833377e+09 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1564098447561264 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.510161817073822 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212962746620178¨ÿ2Öžê‚(:“ ƯæÔóêäÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚150.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.179199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ãÛKë‚(:À ƯæÔ‘Á¹Ø disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0( proc.memory.availableMB‚ 952.54296875* proc.memory.percent‚17.553353309631348 network.sent‚ 7.692507e+06 network.recv‚ 1.050697e+06 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚11386.31005859375% disk./.usageGB‚1110.2250480651855 disk.rbd2.out‚ 64.3115234375 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚11386.31005859375 disk.rbd3.out‚0$ memory_percent‚94.18613910675049# proc.memory.rssMB‚ 2875.94140625( disk.nvme0c0n1.out‚11386.31005859375) disk./.usagePercent‚74.50813451651686 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 109.5546875 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0 disk.dm-1.out‚ 109.5546875 disk.rbd1.out‚ 1.47021484375 disk.sda.out‚ 109.5546875 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 cpu‚25.38466301560263uh‘d2ì- _wandbruntime‚2864  _runtime‚2864|cuÇíÁ & lr-AdamW/pg1‚0.0001623495647668017 & lr-AdamW/pg2‚0.0001623495647668017 trainer/global_step‚3649 % _timestamp‚1.7884507728194451e+09 _runtime‚2864.373813568 _step‚297µ­B§ÌîÆ _runtime‚2864.373813568 _step‚297 & lr-AdamW/pg1‚0.0001623495647668017 & lr-AdamW/pg2‚0.0001623495647668017 trainer/global_step‚3649 % _timestamp‚1.7884507728194451e+09©.2ï- _wandbruntime‚2864  _runtime‚2864NङÜ'ðÖO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212666361721344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3060772819456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210136453138142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53208332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3060772819456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48628589 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚3649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212666379724288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944„Mh‡á'ñÛO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚3649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212666379724288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212666361721344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3060772819456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210136453138142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53208332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3060772819456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48628589 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207086297515008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2865.042989658 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3649 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215784307007488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 ZFDeviceStatsMonitor.on_train_batch_end/alloc“Ö+eated_bytes.large_pool.freed‚212722714760192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235479935504384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210192173703874 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207141201645568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31056326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0}o+¥ì&öæM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31056326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215784981386240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2865.042989658 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3649 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53222346 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215784307007488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17585313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212722714760192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3061592247296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235479935504384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210192173703874 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207141201645568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0«²¥02÷- _wandbruntime‚2865  _runtime‚2865«N•5WøÑ % _timestamp‚1.7884507734898412e+09 /loss/FRt3/detection_loss‚1.9860173319616625 1loss/SRqqt1/assignment_loss‚1.399714174541704 2loss/SRqqt4/assignment_loss‚2.1346802972257137 /loss/FRt1/detection_loss‚2.3715549283752253 /loss/FBt2/detection_loss‚0.6625940747126707 /loss/FRt1/assignment_loss‚2.615603669016969 2loss/SRqqt2/assignment_loss‚0.8714080986729453 0loss/FBt3/assignment_loss‚0.5847137759306597 0loss/SRqqt2/detection_loss‚1.131701059048138 trainer/global_step‚3649 /loss/FRt3/assignment_loss‚3.619525317222841 1loss/FBt1/assignment_loss‚0.42237973809242246 1loss/FBt2/assignment_loss‚0.24821571631760364 /loss/FBt3/detection_loss‚2.5822359451225827 /loss/FRt4/assignment_loss‚5.124432952315719 0loss/FBt4/assignment_loss‚0.7838528776541345 .loss/FRt2/detection_loss‚1.921711382361091 0loss/SRqqt1/detection_loss‚2.019442007896748 1loss/SRqqt3/detection_loss‚1.8064833882131748 0loss/SRqqt4/detection_loss‚2.683327715843916 -loss/FBt4/detection_loss‚3.09939101934433 &loss/total_loss‚46.400370023086666 /loss/FRt2/assignment_loss‚1.127956173857864 2loss/SRqqt3/assignment_loss‚1.4528565743405903 /loss/FRt4/detection_loss‚2.9752230224785983 /loss/FBt1/detection_loss‚2.7753487825393677 epoch‚3 _runtime‚2865.044135027 _step‚300’Ac"\ùÖ /loss/FRt1/assignment_loss‚2.615603669016969 2loss/SRqqt2/assignment_loss‚0.8714080986729453 0loss/FBt3/assignment_loss‚0.5847137759306597 0loss/SRqqt2/detection_loss‚1.131701059048138 trainer/global_step‚3649 /loss/FRt3/assignment_loss‚3.619525317222841 1loss/FBt1/assignment_loss‚0.42237973809242246 1loss/FBt2/assignment_loss‚0.24821571631760364 /loss/FBt3/detection_loss‚2.5822359451225827 /loss/FRt4/assignment_loss‚5.124432952315719 0loss/FBt4/assignment_loss‚0.7838528776541345 .loss/FRt2/detection_loss‚1.921711382361091 0loss/SRqqt1/detection_loss‚2.019442007896748 1loss/SRqqt3/detection_loss‚1.8064833882131748 0loss/SRqqt4/detection_loss‚2.683327715843916 -loss/FBt4/detection_loss‚3.09939101934433 &loss/total_loss‚46.400370023086666 /loss/FRt2/assignment_loss‚1.127956173857864 2loss/SRqqt3/assignment_loss‚1.4528565743405903 /loss/FRt4/detection_loss‚2.9752230224785983 /loss/FBt1/detection_loss‚2.7753487825393677 epoch‚3 _runtime‚2865.044135027 _step‚300 % _timestamp‚1.7884507734898412e+09 /loss/FRt3/detection_loss‚1.9860173319616625 1loss/SRqqt1/assignment_loss‚1.399714174541704 2loss/SRqqt4/assignment_loss‚2.1346802972257137 /loss/FRt1/detection_loss‚2.3715549283752253 /loss/FBt2/detection_loss‚0.6625940747126707¬؇“žú‚(:“ ä¯æÔ’àšÖ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.227199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?R•Iû‚(:¾ ä¯æÔß¡ñØ disk.rbd1.out‚ 1.533203125 disk.sda3.in‚ 0.01171875# memory_percent‚94.1777229309082) disk./.usagePercent‚74.50799985655387 disk.drbd1777.in‚0 cpu‚25.381286026527988 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 65.9638671875( disk.nvme0c0n1.out‚11957.47705078125& proc.memory.availableMB‚ 953.921875" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 network.recv‚ 1.062924e+06 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚11957.47705078125 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 network.sent‚ 7.914174e+06% disk./.usageGB‚1110.2230415344238 disk.rbd3.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚11957.47705078125# proc.memory.rssMB‚ 2874.95703125 disk.dm-1.out‚ 110.59765625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 110.59765625 disk.sda3.out‚ 110.59765625) proc.memory.percent‚17.54734516143799 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875F7x‘ü‚(:† ó¯æÔ¯ŸÄ× gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë€fº:ý‚(:¯ ó¯æÔ„ÒöÙ disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 953.7734375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.rbd0.out‚0 cpu‚25.37947998709304 proc.cpu.threads‚17 disk.sda3.out‚ 111.11328125# disk.nvme0c0n1.out‚ 11958.140625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 network.sent‚ 8.01143e+06 network.recv‚ 1.068533e+06% disk./.usageGB‚1110.2230491638184 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.533203125 disk.rbd2.out‚ 65.9794921875" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda1.in‚0 disk.sda.out‚ 111.11328125 disk.rbd0.in‚0 disk.sda3.in‚ 0.01171875) disk./.usagePercent‚74.50800036856893 disk.dm-0.out‚ 11958.140625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 111.11328125) proc.memory.percent‚17.54734516143799! disk.nvme0n1.out‚ 11958.140625 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚94.17862892150879# proc.memory.rssMB‚ 2874.95703125ž‚:2þ- _wandbruntime‚2899  _runtime‚2899 ªÜ2Éÿà _step‚301 ' lr-AdamW/pg1‚0.00016148080241768235 ' lr-AdamW/pg2‚0.00016148080241768235 trainer/global_step‚3699 % _timestamp‚1.7884508081480396e+09 _runtime‚2899.702431918¡»šÎ€È % _timestamp‚1.7884508081480396e+09 _runtime‚2899.702431918 _step‚301 ' lr-AdamW/pg1‚0.00016148080241768235 ' lr-AdamW/pg2‚0.00016148080241768235 trainer/global_step‚3699­ž6ÇL2- _wandbruntime‚2899  _runtime‚2899ó,×Ý'‚×O YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212922495005550 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3255721034240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3090990962542 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 trainer/global_step‚3699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238542785055744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123817281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209831504043008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215484031666688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69908151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69908151 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215484013663744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218586446654464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3090336524786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Ã׌׃ÜO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215484013663744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3090336524786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212922495005550 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3255721034240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235286898300416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3090990962542 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 trainer/global_step‚3699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238542785055744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235286874360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123817281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53909132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209831504043008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215484031666688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69908151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69908151 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209831487003648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215484013663744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218586446654464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414®Å o2„- _wandbruntime‚2900  _runtime‚2900ö—áFç&…áM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69926605 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884508088135507e+09 _runtime‚2900.368358632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _step‚303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53923148 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215540366702592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212978215571282 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235348525097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3256745977856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218643619134464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 Ê ì&†æM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235348525097984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3256745977856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69946669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218643619134464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69926605 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884508088135507e+09 _runtime‚2900.368358632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _step‚303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53923148 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215540366702592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212978215571282 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215540384705536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750¯p²•%2‡- _wandbruntime‚2900  _runtime‚2900È>À•ZˆÔ 0loss/FRt2/assignment_loss‚1.5172301086328797 /loss/FRt1/detection_loss‚2.8464946114655696 % _timestamp‚1.7884508088155496e+09 &loss/total_loss‚47.493803900148265 /loss/FRt1/assignment_loss‚3.745289472016421 0loss/FBt1/assignment_loss‚0.6144881776999682 0loss/FBt3/assignment_loss‚0.4355359518279632 .loss/FRt3/detection_loss‚2.072770498845042 /loss/FBt2/detection_loss‚0.5307876378103908 1loss/FBt4/assignment_loss‚0.25609040395665944 /loss/FRt2/detection_loss‚1.7057944258390847 1loss/SRqqt3/detection_loss‚1.8226703525857721 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚0.9789163489037982 2loss/SRqqt3/assignment_loss‚1.0258550715256245 epoch‚3 /loss/FRt4/assignment_loss‚4.615477121793306 1loss/FBt2/assignment_loss‚0.23965076989048933 .loss/FRt4/detection_loss‚3.162076908808488 .loss/FBt4/detection_loss‚2.686762309074402 1loss/SRqqt1/detection_loss‚1.7927234305275812 0loss/SRqqt4/detection_loss‚2.327005212626806 /loss/FBt3/detection_loss‚2.6367525309324265 1loss/SRqqt1/assignment_loss‚1.835194457805267 /loss/FBt1/detection_loss‚3.3971113711595535 _runtime‚2900.377751652 _step‚304 0loss/FRt3/assignment_loss‚3.7813318298787486 1loss/SRqqt4/assignment_loss‚2.270201976706342 1loss/SRqqt2/detection_loss‚1.1975929198356776ÔªIÊ_‰Ù 1loss/SRqqt1/assignment_loss‚1.835194457805267 /loss/FBt1/detection_loss‚3.3971113711595535 _runtime‚2900.377751652 _step‚304 0loss/FRt3/assignment_loss‚3.7813318298787486 1loss/SRqqt4/assignment_loss‚2.270201976706342 1loss/SRqqt2/detection_loss‚1.1975929198356776 0loss/FRt2/assignment_loss‚1.5172301086328797 /loss/FRt1/detection_loss‚2.8464946114655696 % _timestamp‚1.7884508088155496e+09 &loss/total_loss‚47.493803900148265 /loss/FRt1/assignment_loss‚3.745289472016421 0loss/FBt1/assignment_loss‚0.6144881776999682 0loss/FBt3/assignment_loss‚0.4355359518279632 .loss/FRt3/detection_loss‚2.072770498845042 /loss/FBt2/detection_loss‚0.5307876378103908 1loss/FBt4/assignment_loss‚0.25609040395665944 /loss/FRt2/detection_loss‚1.7057944258390847 1loss/SRqqt3/detection_loss‚1.8226703525857721 trainer/global_step‚3699 2loss/SRqqt2/assignment_loss‚0.9789163489037982 2loss/SRqqt3/assignment_loss‚1.0258550715256245 epoch‚3 /loss/FRt4/assignment_loss‚4.615477121793306 1loss/FBt2/assignment_loss‚0.23965076989048933 .loss/FRt4/detection_loss‚3.162076908808488 .loss/FBt4/detection_loss‚2.686762309074402 1loss/SRqqt1/detection_loss‚1.7927234305275812 0loss/SRqqt4/detection_loss‚2.327005212626806 /loss/FBt3/detection_loss‚2.6367525309324265°þ ÎužŠ‚(:“ ‚°æÔâ˾Õ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.623' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.724599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ç‚(‹‚(:Á ‚°æÔüî…Ø# disk.dm-0.oÏ… $ut‚12055.97412109375 disk.dm-1.out‚ 111.51171875* proc.memory.percent‚17.064857482910156 cpu‚25.377830264206263 disk.rbd1.out‚ 1.533203125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0 network.recv‚ 1.072588e+06 disk.sda2.in‚0) proc.memory.availableMB‚ 1032.05859375 disk.sda3.out‚ 111.51171875 disk.rbd3.out‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2795.90625) disk./.usagePercent‚74.50799217632783( disk.nvme0c0n1.out‚12055.97412109375 proc.cpu.threads‚17% disk./.usageGB‚1110.2229270935059 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625& disk.nvme0n1.out‚12055.97412109375$ memory_percent‚93.70081424713135 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.out‚0 network.sent‚ 8.017624e+06 disk.rbd1.in‚0 disk.sda.out‚ 111.51171875 disk.rbd2.out‚66.06591796875nD|“Œ‚(:ˆ ‘°æÔœý¶Õ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M…>gB‚(:· ‘°æÔ´‹öØ disk.dm-0.out‚12056.87890625) disk./.usagePercent‚74.50797220774015 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚66.08154296875* proc.memory.percent‚17.064857482910156 disk.dm-1.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625% disk.nvme0c0n1.out‚12056.87890625 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17 network.sent‚ 8.125844e+06 disk.sda.out‚ 112.08984375 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1032.35546875 disk.dm-1.out‚ 112.08984375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 cpu‚25.37614821686287% disk./.usageGB‚1110.2226295471191 disk.drbd1777.out‚0$ memory_percent‚93.69900226593018 network.recv‚ 1.078398e+06 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0# disk.nvme0n1.out‚12056.87890625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 112.08984375Íù2Ž- _wandbruntime‚2935  _runtime‚2935hΔÇÁ & lr-AdamW/pg2‚0.0001605975900048404 trainer/global_step‚3749 % _timestamp‚1.7884508438017652e+09 _runtime‚2935.356138995 _step‚305 & lr-AdamW/pg1‚0.0001605975900048404– âæÌÆ & lr-AdamW/pg2‚0.0001605975900048404 trainer/global_step‚3749 % _timestamp‚1.7884508438017652e+09 _runtime‚2935.356138995 _step‚305 & lr-AdamW/pg1‚0.0001605975900048404±UuË#2‘- _wandbruntime‚2935  _runtime‚2935°KW!Ý'’×O JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 _step‚306 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49895613 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49895175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3143400793600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218301665606144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212576710571008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31864958 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298491593728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221444410037248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3131171864194 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3142744431104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2935.357083214 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212576693531648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221445084402688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54609930 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ȱ»ïâ'“ÜO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70854241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70834203 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215708536872958 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125444133 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238369435183616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 _step‚306 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49895613 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49895175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3143400793600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218301665606144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218301683609088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212576710571008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54609932 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31864958 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298491593728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221444410037248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215707865395842 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3131171864194 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3142744431104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54609930 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2935.357083214 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212576693531648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221445084402688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚218301665606144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54609930 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54623946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788450844474321e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54623948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125476603 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238431061981184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215764257438690 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212631614701568 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18034642 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3299346749952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 _runtime‚2936.029138097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54623948 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool¹p].allocated‚31873126 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54623946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18035083 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788450844474321e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54623948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚3749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125476603 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238431085921280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218358036647936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238431061981184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215764257438690 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212631614701568 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18034642 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3299346749952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 _runtime‚2936.029138097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54623948 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31873126 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71760255 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3183730236928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3184386599424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3184386599424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221119317548544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3172661641358 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32273356 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2970.949601986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273358 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 $ _timestamp‚1.788450879395025e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872P°Bî.¥ÛO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221119317548544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3172661641358 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚3799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32273356 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚2970.949601986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273358 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 $ _timestamp‚1.788450879395025e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 o¦‚÷³?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71760255 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3183730236928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3341395488768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3184386599424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3341571695616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3184386599424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55310732 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928¶—8t2¦- _wandbruntime‚2971  _runtime‚2971n#šÜç&§áM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224360894630912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224360220252160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221175670587392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚311 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3342413927936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221175688590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71798773 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215376804190208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224360894630912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2971.614436654 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221175688590336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55324748 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71798773 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215376821229568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71778709 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215376804190208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71778709 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221175688590336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18261122 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224360894630912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3184549664768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚2971.614436654 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752·r”ˆÏ2©- _wandbruntime‚2971  _runtime‚2971ÅÃFvUªÏ .loss/FRt4/detection_loss‚3.059288283189138 /loss/FBt3/detection_loss‚2.0883910904328027 /loss/FBt4/assignment_loss‚0.384764350104061 .loss/FRt1/detection_loss‚2.433972570883191 2loss/SRqqt2/assignment_loss‚1.3788679339922965 1loss/SRqqt3/assignment_loss‚2.030656745819294 0loss/FBt3/assignment_loss‚0.4333178314069907 _step‚312 /loss/FRt3/assignment_loss‚3.419995032250881 0loss/FBt1/assignment_loss‚0.6833481209145652 %loss/total_loss‚46.61583583102812 trainer/global_step‚3799 1loss/SRqqt1/assignment_loss‚1.660957789621674 /loss/FRt3/detection_loss‚2.4963699967289963 1loss/SRqqt1/detection_loss‚1.8021698548243597 1loss/SRqqt3/detection_loss‚1.9427585546737132 1loss/SRqqt4/detection_loss‚2.7281377870461037 /loss/FBt1/detection_loss‚2.4641175071398416 0loss/FRt2/assignment_loss‚1.9799433337605519 0loss/FRt4/assignment_loss‚3.6178365697463355 /loss/FBt4/detection_loss‚2.6549299521879717 0loss/FBt2/assignment_loss‚0.1503020927052555 1loss/SRqqt2/detection_loss‚1.0744648567633703 .loss/FBt2/detection_loss‚0.721045328388076 % _timestamp‚1.7884508800613928e+09 /loss/FRt2/detection_loss‚2.0247585679525915 -loss/FRt1/assignment_loss‚2.7284089728542 2loss/SRqqt4/assignment_loss‚2.6570327076418647 epoch‚3 _runtime‚2971.615702409ƒîY•Z«Ô 0loss/FBt2/assignment_loss‚0.1503020927052555 1loss/SRqqt2/detection_loss‚1.0744648567633703 .loss/FBt2/detection_loss‚0.721045328388076 % _timestamp‚1.7884508800613928e+09 /loss/FRt2/detection_loss‚2.0247585679525915 -loss/FRt1/assignment_loss‚2.7284089728542 2loss/SRqqt4/assignment_loss‚2.6570327076418647 epoch‚3 _runtime‚2971.615702409 .loss/FRt4/detection_loss‚3.059288283189138 /loss/FBt3/detection_loss‚2.0883910904328027 /loss/FBt4/assignment_loss‚0.384764350104061 .loss/FRt1/detection_loss‚2.433972570883191 2loss/SRqqt2/assignment_loss‚1.3788679339922965 1loss/SRqqt3/assignment_loss‚2.030656745819294 0loss/FBt3/assignment_loss‚0.4333178314069907 _step‚312 /loss/FRt3/assignment_loss‚3.419995032250881 0loss/FBt1/assignment_loss‚0.6833481209145652 %loss/total_loss‚46.61583583102812 trainer/global_step‚3799 1loss/SRqqt1/assignment_loss‚1.660957789621674 /loss/FRt3/detection_loss‚2.4963699967289963 1loss/SRqqt1/detection_loss‚1.8021698548243597 1loss/SRqqt3/detection_loss‚1.9427585546737132 1loss/SRqqt4/detection_loss‚2.7281377870461037 /loss/FBt1/detection_loss‚2.4641175071398416 0loss/FRt2/assignment_loss‚1.9799433337605519 0loss/FRt4/assignment_loss‚3.6178365697463355 /loss/FBt4/detection_loss‚2.6549299521879717¸«àH“¬‚(:ˆ Í°æÔÄËÜÕ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÏC­‚(:¸ Í°æÔßëªØ disk.drbd1777.out‚0 proc.memory.rssMB‚ 2795.90625 cpu‚25.369575391815868 disk.rbd1.out‚ 1.533203125 disk.rbd2.in‚0 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875) disk./.usagePercent‚74.50800113659153 disk.rbd2.out‚67.87451171875 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.out‚ 114.12890625 proc.cpu.threads‚17 disk.sda1.in‚0! disk.dm-0.out‚12350.611328125 disk.sda.out‚ 114.12890625' proc.memory.availableMB‚ 557.0390625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875* proc.memory.percent‚17.064857482910156 network.sent‚ 8.252151e+06 disk.sda3.out‚ 114.12890625" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚12350.611328125 network.recv‚ 1.096748e+06% disk./.usageGB‚1110.2230606079102 disk.rbd0.out‚0# memory_percent‚96.6001033782959 disk.sda1.out‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚12350.611328125 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0yÖƒž®‚(:“ Ü°æÔ覫Õ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.942' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.188399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n­P ?¯‚(:´ Ü°æÔËÌïØ disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 114.89453125 disk.sda1.out‚0( proc.memory.availableMB‚ 556.79296875 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚ 67.9609375$ memory_percent‚96.60160541534424 cpu‚25.36789924767212 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚12350.904296875 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 disk.sda.out‚ 114.89453125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 114.89453125" disk.nvme0c0n1.in‚ 162.94140625$ disk.nvme0n1.out‚12350.904296875 network.sent‚ 8.361173e+06) disk./.usagePercent‚74.50800190461413 disk.dm-0.in‚ 162.94140625 network.recv‚ 1.103169e+06$ disk./.usageGB‚1110.223072052002! disk.dm-0.out‚12350.904296875 disk.rbd1.out‚ 1.533203125 disk.rbd0.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625SÝ{¢2°- _wandbruntime‚3006  _runtime‚3006¡ÇŸÉ±Ã ' lr-AdamW/pg1‚0.00015878855420157464 ' lr-AdamW/pg2‚0.00015878855420157464 trainer/global_step‚3849 % _timestamp‚1.7884509150831594e+09 _runtime‚3006.637606076 _step‚313ÇÃ? Î²È ' lr-AdamW/pg1‚0.00015878855420157464 ' lr-AdamW/pg2‚0.00015878855420157464 trainer/global_step‚3849 % _timestamp‚1.7884509150831594e+09 _runtime‚3006.637606076 _step‚313¹'ªâè2³- _wandbruntime‚3006  _runtime‚3006ºzI´×O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_poolRì\”'.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227162359899136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247918856526336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72686307 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227161685533696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51165485 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212842543010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884509150839188e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223936987493888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 trainer/global_step‚3849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56011530 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56011532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56011530 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244534508950016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224716042752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227161685533696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51165485 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212842543010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884509150839188e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223936987493888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 trainer/global_step‚3849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56011530 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56011532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56011530 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244534508950016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224716042752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18484163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72724825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247981367677440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚315 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225535470592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247981175932416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385207989760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128730307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221336341173506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214313415938 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18488666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223993322529792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227219532379136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3385040184832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3226191846400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218122027757568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223993322529792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227219532379136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223993340532736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ƒ:çœ 3385040184832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3226191846400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218122027757568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221335669696182 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72724825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247981367677440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚315 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225535470592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247981175932416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223993322529792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385207989760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128730307»Ž”¤ž2¹- _wandbruntime‚3007  _runtime‚3007´.œóWºÑ 2loss/SRqqt4/assignment_loss‚2.6324608432395116 /loss/FRt2/detection_loss‚1.8711153030395509 .loss/FRt3/detection_loss‚2.272404956263165 1loss/SRqqt4/detection_loss‚2.7659586167761256 _step‚316 /loss/FRt3/assignment_loss‚3.979618005974348 0loss/FBt3/assignment_loss‚0.5280333978589624 /loss/FRt4/detection_loss‚3.4461379274725914 1loss/SRqqt2/detection_loss‚1.1644319045756544 1loss/SRqqt1/assignment_loss‚1.170838014415975 1loss/SRqqt1/detection_loss‚1.6893794549242505 /loss/FBt2/detection_loss‚0.6706373402137461 /loss/FBt4/detection_loss‚3.1116940081119537 1loss/FBt2/assignment_loss‚0.28562661205608875 .loss/FRt1/detection_loss‚2.583830112634703 epoch‚3 # _timestamp‚1.78845091576246e+09 2loss/SRqqt2/assignment_loss‚1.2449045238218137 .loss/FBt3/detection_loss‚3.004909267028173 %loss/total_loss‚50.07718588675061 0loss/FRt2/assignment_loss‚1.9053242531689731 1loss/FBt1/assignment_loss‚0.35357431464009487 -loss/FBt1/detection_loss‚3.65993286882128 _runtime‚3007.316782528 /loss/FRt1/assignment_loss‚2.644800956859145 2loss/SRqqt3/assignment_loss‚1.9515336199104787 1loss/SRqqt3/detection_loss‚1.8269134502112865 1loss/FBt4/assignment_loss‚0.43865804658271373 trainer/global_step‚3849 /loss/FRt4/assignment_loss‚4.874468088150024‘È,œ\»Ö /loss/FRt1/assignment_loss‚2.644800956859145 2loss/SRqqt3/assignment_loss‚1.9515336199104787 1loss/SRqqt3/detection_loss‚1.8269134502112865 1loss/FBt4/assignment_loss‚0.43865804658271373 trainer/global_step‚3849 /loss/FRt4/assignment_loss‚4.874468088150024 2loss/SRqqt4/assignment_loss‚2.6324608432395116 /loss/FRt2/detection_loss‚1.8711153030395509 .loss/FRt3/detection_loss‚2.272404956263165 1loss/SRqqt4/detection_loss‚2.7659586167761256 _step‚316 /loss/FRt3/assignment_loss‚3.979618005974348 0loss/FBt3/assignment_loss‚0.5280333978589624 /loss/FRt4/detection_loss‚3.4461379274725914 1loss/SRqqt2/detection_loss‚1.1644319045756544 1loss/SRqqt1/assignment_loss‚1.170838014415975 1loss/SRqqt1/detection_loss‚1.6893794549242505 /loss/FBt2/detection_loss‚0.6706373402137461 /loss/FBt4/detection_loss‚3.1116940081119537 1loss/FBt2/assignment_loss‚0.28562661205608875 .loss/FRt1/detection_loss‚2.583830112634703 epoch‚3 # _timestamp‚1.78845091576246e+09 2loss/SRqqt2/assignment_loss‚1.2449045238218137 .loss/FBt3/detection_loss‚3.004909267028173 %loss/total_loss‚50.07718588675061 0loss/FRt2/assignment_loss‚1.9053242531689731 1loss/FBt1/assignment_loss‚0.35357431464009487 -loss/FBt1/detection_loss‚3.65993286882128 _runtime‚3007.316782528¼íã•£“¼‚(:ˆ ë°æÔЖÚÖ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Ì”8A½‚(:¶ ë°æÔð¬ÌØ network.sent‚ 8.366393e+06 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda2.in‚0# memory_percent‚96.6017484664917( proc.memory.availableMB‚ 556.76953125* proc.memory.percent‚17.064857482910156 disk.rbd3.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625% disk.nvme0n1.out‚12448.5166015625 network.recv‚ 1.106521e+06 disk.sda1.in‚0 proc.memory.rssMB‚ 2795.90625 disk.sda2.out‚0 disk.rbd1.out‚ 1.533203125 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625" disk.dm-0.out‚12448.5166015625 disk.rbd3.out‚0 disk.sda.out‚ 115.21484375 disk.rbd0.in‚0 disk.rbd2.out‚ 67.9765625 disk.drbd1777.in‚0 disk.dm-1.out‚ 115.21484375 proc.cpu.threads‚17 disk.sda3.out‚ 115.21484375) disk./.usagePercent‚74.50800958484017# disk./.usageGB‚1110.22318649292 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚12448.5166015625 cpu‚25.366335512116873oCG‘¾‚(:† ú°æÔç·¿Õ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í„Ð1¿‚(:¦ ú°æÔ¾ŽþØ network.recv‚ 1.112821e+06" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda3.out‚115.75* proc.memory.percent‚17.064857482910156 disk.sda.out‚115.75 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.rbd3.out‚0 network.sent‚ 8.475335e+06% disk./.usageGB‚1110.2223358154297 disk.dm-0.in‚ 162.94140625 disk.dm-1.out‚115.75" disk.dm-0.out‚12448.8173828125 disk.sda3.in‚ 0.01171875$ memory_percent‚96.60172462463379 disk.sda1.in‚0) disk./.usagePercent‚74.50795249516003 disk.rbd2.out‚ 70.25390625' disk.nvme0c0n1.out‚12448.8173828125 disk.rbd1.in‚0 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.in‚0% disk.nvme0n1.out‚12448.8173828125 disk.sda2.out‚0 cpu‚25.36485816599587 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.533203125' proc.memory.availableMB‚ 556.7734375]*³à2À- _wandbruntime‚3042  _runtime‚3042H¤¬¹ÈÁ trainer/global_step‚3899 $ _timestamp‚1.788450950651816e+09 _runtime‚3042.206218515 _step‚317 ' lr-AdamW/pg1‚0.00015786310781150078 ' lr-AdamW/pg2‚0.00015786310781150078l(EÍÂÇ _step‚317 ' lr-AdamW/pg1‚0.00015786310781150078 ' lr-AdamW/pg2‚0.00015786310781150078 trainer/global_step‚3899 $ _timestamp‚1.788450950651816e+09 _runtime‚3042.206218515½)]*ª2Ã- _wandbruntime‚3042  _runtime‚3042åmúVÝ'Ä×O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130324689 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247617021893120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56712332 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230020997647360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33090158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3042.207705607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3266358211072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 % _timestamp‚1.7884509506528003e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73612359 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 trainer/global_step‚3899 _step‚318 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73632397 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3265701848576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254332320174 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56712330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18711371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116`UÓ±!ÅÜO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3042.207705607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3266358211072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 % _timestamp‚1.7884509506528003e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73612359 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51801529 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 trainer/global_step‚3899 _step‚318 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73632397 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3265701848576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254332320174 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56712330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56712330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18711371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130324689 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247617021893120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56712332 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230020997647360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3266358211072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33090158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752¾µÄu2Æ- _wandbruntime‚3042  _runtime‚3042·Ä‘æ&ÇàM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3254494317382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251106915701760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220867234285568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 trainer/global_step‚3899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788450951322735e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3266521276416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73650877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247678696570880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3428045034496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3254494317382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251106915701760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220867234285568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247678672630784 trainer/global_step‚3899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56726346 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788450951322735e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56726348 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3266521276416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73650877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247678696570880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430¿Ph Î2É- _wandbruntime‚3042  _runtime‚3042f\h'UÊÏ 0loss/FRt1/assignment_loss‚3.1792837407277976 /loss/FRt1/detection_loss‚2.3396279747071476 trainer/global_step‚3899 2loss/SRqqt4/assignment_loss‚1.7415374630579241 0loss/FBt4/assignment_loss‚0.8315607820238385 -loss/FBt3/detection_loss‚2.57245022803545 /loss/FRt4/detection_loss‚3.7069470008214314 /loss/FBt4/detection_loss‚3.0086471693856374 _step‚320 2loss/SRqqt2/assignment_loss‚1.2331584359570924 .loss/FRt3/detection_loss‚2.171896514567462 1loss/SRqqt1/detection_loss‚1.9351837616793963 /loss/FRt2/assignment_loss‚2.292468637228012 /loss/FRt3/assignment_loss‚3.109469509124756 /loss/FRt4/assignment_loss‚4.631533432006836 .loss/FBt1/detection_loss‚2.747036717154763 %loss/total_loss‚48.86570368055223 epoch‚3 2loss/SRqqt1/assignment_loss‚1.5111048285450255 1loss/FBt2/assignment_loss‚0.25206177275825387 0loss/FBt3/assignment_loss‚0.9035833826783346 /loss/SRqqt3/detection_loss‚2.04783581164868 1loss/SRqqt4/detection_loss‚2.6733695886753224 1loss/SRqqt3/assignment_loss‚1.594867075631476 /loss/FRt2/detection_loss‚1.9461680897153342 0loss/SRqqt2/detection_loss‚1.179658651144968 % _timestamp‚1.7884509513253515e+09 _runtime‚3042.879678551 0loss/FBt1/assignment_loss‚0.5633576318009132 /loss/FBt2/detection_loss‚0.6928954814763791”_¨;ZËÔ /loss/FRt4/detection_loss‚3.7069470008214314 /loss/FBt4/detection_loss‚3.0086471693856374 _step‚320 2loss/SRqqt2/assignment_loss‚1.2331584359570924 .loss/FRt3/detection_loss‚2.171896514567462 1loss/SRqqt1/detection_loss‚1.9351837616793963 /loss/FRt2/assignment_loss‚2.292468637228012 /loss/FRt3/assignment_loss‚3.109469509124756 /loss/FRt4/assignment_loss‚4.631533432006836 .loss/FBt1/detection_loss‚2.747036717154763 %loss/total_loss‚48.86570368055223 epoch‚3 2loss/SRqqt1/assignment_loss‚1.5111048285450255 1loss/FBt2/assignment_loss‚0.25206177275825387 0loss/FBt3/assignment_loss‚0.9035833826783346 /loss/SRqqt3/detection_loss‚2.04783581164868 1loss/SRqqt4/detection_loss‚2.6733695886753224 1loss/SRqqt3/assignment_loss‚1.594867075631476 /loss/FRt2/detection_loss‚1.9461680897153342 0loss/SRqqt2/detection_loss‚1.179658651144968 % _timestamp‚1.7884509513253515e+09 _runtime‚3042.879678551 0loss/FBt1/assignment_loss‚0.5633576318009132 /loss/FBt2/detection_loss‚0.6928954814763791 0loss/FRt1/assignment_loss‚3.1792837407277976 /loss/FRt1/detection_loss‚2.3396279747071476 trainer/global_step‚3899 2loss/SRqqt4/assignment_loss‚1.7415374630579241 0loss/FBt4/assignment_loss‚0.8315607820238385 -loss/FBt3/detection_loss‚2.57245022803545À”Î: “Ì‚(:ˆ ‰±æÔÇþÃÕ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ÌO<Í‚(:± ‰±æÔƒð˜Ù$ memory_percent‚96.59318923950195 network.sent‚ 8.480861e+06 disk.sda.out‚ 116.265625 disk.sda1.out‚0& proc.memory.availableMB‚ 558.171875" disk.dm-0.out‚12546.5263671875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 70.26953125 disk.sda2.out‚0 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625% disk.nvme0n1.out‚12546.5263671875% disk./.usageGB‚1110.2232704162598 disk.rbd0.out‚0 network.recv‚ 1.116942e+06" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd3.out‚0 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 cpu‚25.363316342343776 proc.cpu.threads‚17( disk./.usagePercent‚74.5080152170059 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 disk.sda3.out‚ 116.265625' disk.nvme0c0n1.out‚12546.5263671875 disk.drbd1777.in‚0 disk.dm-1.out‚ 116.265625ª,zžÎ‚(:“ ˜±æÔ™üÑÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.003199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*AèAEÏ‚(:º ˜±æÔ•Æ«Ø( disk./.usagePercent‚74.5080113768929 disk.rbd0.out‚0 network.recv‚ 1.123629e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625 disk.sda3.out‚ 116.8828125 disk.rbd2.in‚0 disk.dm-0.in‚ 162.94140625 disk.sda.out‚ 116.8828125 cpu‚25.361807682906086 proc.cpu.threads‚17( disk.nvme0c0n1.out‚12546.93017578125 disk.sda2.in‚0 disk.dm-1.out‚ 116.8828125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0$ memory_percent‚96.59490585327148% disk./.usageGB‚1110.2232131958008 disk.nvme0n1.in‚ 162.94140625# disk.dm-0.out‚12546.93017578125 disk.rbd3.out‚0 network.sent‚ 8.589905e+06 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875* proc.memory.percent‚17.064857482910156 disk.sda1.in‚0 disk.rbd2.out‚70.35595703125& disk.nvme0n1.out‚12546.93017578125 disk.rbd1.out‚ 1.533203125& proc.memory.availableMB‚ 557.890625 proc.memory.rssMB‚ 2795.90625zž_žÐ‚(:“ §±æÔÙÃÞÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.762' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.152400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©253Ñ‚(:¨ §±æÔ‘½­Ø) disk./.usagePercent‚74.50801931312647 disk.sda.out‚ 117.265625 disk.drbd1777.in‚0 cpu‚25.360218327102174 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.out‚ 117.265625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875# memory_percent‚96.5919017791748 network.sent‚ 8.594989e+06 disk.rbd3.out‚0 disk.rbd0.in‚0! disk.nvme0n1.out‚ 12644.828125 disk.drbd1777.out‚0 disk.dm-0.out‚ 12644.828125 disk.sda2.in‚0' proc.memory.availableMB‚ 558.3828125 network.recv‚ 1.126517e+06 disk.rbd1.out‚ 1.533203125 disk.rbd3.in‚0 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 disk.nvme0n1.in‚ 162.94140625 disk.sda3.out‚ 117.265625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.out‚70.37158203125$ disk./.usageGB‚1110.223331451416 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 12644.828125 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0G6>Z2Ò- _wandbruntime‚3078  _runtime‚3078®ô´VÇÓÁ _step‚321 & lr-AdamW/pg1‚0.0001569239652798379 & lr-AdamW/pg2‚0.0001569239652798379 trainer/global_step‚3949 % _timestamp‚1.7884509864924355e+09 _runtime‚3078.046843327f´¾1ÌÔÆ _runtime‚3078.046843327 _step‚321 & lr-AdamW/pg1‚0.0001569239652798379 & lr-AdamW/pg2‚0.0001569239652798379 trainer/global_step‚3949 % _timestamp‚1.7884509864924355e+09Á¼÷T@2Õ- _wandbruntime‚3078  _runtime‚3078‹haó ÖÖO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223557519643648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74538411 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚26058­½Má'×ÛO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223557519643648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74538411 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3307344016896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605Â>>2Ø- _wandbruntime‚3078  _runtime‚3078üyÂç&ÙáM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254232162077184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74576929 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3078.715429619 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250761233454080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 % _timestamp‚1.7884509871605713e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232936133496832 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232936807875584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470952563200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57427148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3078.715429619 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33506724 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250761233454080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 % _timestamp‚1.7884509871605713e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232936133496832 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3308163458048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57427146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74576929 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223612440813568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232936807875584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470952563200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57427148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908ÃJtt2Û- _wandbruntime‚3078  _runtime‚3078ȶ¾¯ZÜÔ 2loss/SRqqt4/assignment_loss‚1.9119432299367842 0loss/FBt2/assignment_loss‚0.3260337199565968 1loss/FBt3/assignment_loss‚0.26953642393151916 0loss/FBt4/assignment_loss‚0.3129580013483064 %loss/total_loss‚46.91276492549463 $ _timestamp‚1.788450987162208e+09 0loss/FRt3/assignment_loss‚3.3951630670516217 0loss/FBt1/assignment_loss‚0.4946944296360016 .loss/FRt4/detection_loss‚3.153138749874555 /loss/FBt2/detection_loss‚0.7545878236590279 epoch‚3 /loss/FRt2/assignment_loss‚1.551062302769355 /loss/FRt4/assignment_loss‚4.410646232274862 2loss/SRqqt3/assignment_loss‚1.7705735141145331 /loss/FRt1/detection_loss‚2.3873359628699045 /loss/FRt2/detection_loss‚1.8051681200850684 0loss/SRqqt4/detection_loss‚2.625035971403122 /loss/FBt1/detection_loss‚2.9164944708347322 2loss/SRqqt2/assignment_loss‚1.1779536110299982 .loss/FRt3/detection_loss‚1.997775315261278 /loss/FBt3/detection_loss‚2.0917500336964925 2loss/SRqqt1/assignment_loss‚1.4567601569377038 1loss/SRqqt2/detection_loss‚1.5005472132511306 /loss/FBt4/detection_loss‚3.1911904546949597 trainer/global_step‚3949 0loss/FRt1/assignment_loss‚3.8586173951625824 1loss/SRqqt3/detection_loss‚1.6909349563778664 1loss/SRqqt1/detection_loss‚1.8628637693366226 _runtime‚3078.716774834 _step‚324Üš»_ÝÙ 0loss/FRt3/assignment_loss‚3.3951630670516217 0loss/FBt1/assignment_loss‚0.4946944296360016 .loss/FRt4/detection_loss‚3.153138749874555 /loss/FBt2/detection_loss‚0.7545878236590279 epoch‚3 /loss/FRt2/assignment_loss‚1.551062302769355 /loss/FRt4/assignment_loss‚4.410646232274862 2loss/SRqqt3/assignment_loss‚1.7705735141145331 /loss/FRt1/detection_loss‚2.3873359628699045 /loss/FRt2/detection_loss‚1.8051681200850684 0loss/SRqqt4/detection_loss‚2.625035971403122 /loss/FBt1/detection_loss‚2.9164944708347322 2loss/SRqqt2/assignment_loss‚1.1779536110299982 .loss/FRt3/detection_loss‚1.997775315261278 /loss/FBt3/detection_loss‚2.0917500336964925 2loss/SRqqt1/assignment_loss‚1.4567601569377038 1loss/SRqqt2/detection_loss‚1.5005472132511306 /loss/FBt4/detection_loss‚3.1911904546949597 trainer/global_step‚3949 0loss/FRt1/assignment_loss‚3.8586173951625824 1loss/SRqqt3/detection_loss‚1.6909349563778664 1loss/SRqqt1/detection_loss‚1.8628637693366226 _runtime‚3078.716774834 _step‚324 2loss/SRqqt4/assignment_loss‚1.9119432299367842 0loss/FBt2/assignment_loss‚0.3260337199565968 1loss/FBt3/assignment_loss‚0.26953642393151916 0loss/FBt4/assignment_loss‚0.3129580013483064 %loss/total_loss‚46.91276492549463 $ _timestamp‚1.788450987162208e+09Äbö“Þ‚(:ˆ ¶±æÔ’žõÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.029' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ý86ß‚(:« ¶±æÔ‡¯§Ù" disk.nvme0c0n1.in‚ 162.94140625! disk.nvme0n1.out‚ 12645.203125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 network.sent‚ 8.600668e+06 disk.nvme0n1.in‚ 162.94140625 disk.rbd2.in‚0( proc.memory.availableMB‚ 558.12890625 disk.sda.out‚ 117.953125$ memory_percent‚96.59345149993896 disk.drbd1777.in‚0 disk.dm-0.out‚ 12645.203125 disk.sda2.out‚0 disk.sda3.out‚ 117.953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 117.953125 disk.rbd1.out‚ 1.533203125 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.out‚70.49658203125 disk.dm-0.in‚ 162.94140625 proc.memory.rssMB‚ 2795.90625% disk./.usageGB‚1110.2232971191406# disk.nvme0c0n1.out‚ 12645.203125 disk.drbd1777.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.064857482910156 cpu‚25.358789066414513 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.131233e+06) disk./.usagePercent‚74.50801700905866å˜QÄ?à‚(:´ űæÔБÖØ disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚12742.93017578125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0) disk./.usagePercent‚74.50801956913399 disk.sda3.out‚ 118.34375 disk.rbd0.in‚0 network.recv‚ 1.137387e+06$ memory_percent‚96.59123420715332* proc.memory.percent‚17.064857482910156 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚70.50830078125 disk.dm-0.in‚ 162.94140625 disk.rbd1.out‚ 1.533203125 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 118.34375 network.sent‚ 8.70883e+06% disk./.usageGB‚1110.2233352661133 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 118.34375 disk.rbd1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 558.4921875 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17 disk.rbd3.in‚0& disk.nvme0n1.out‚12742.93017578125 disk.sda2.out‚0# disk.dm-0.out‚12742.93017578125 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625 cpu‚25.35732004978402R¥!žá‚(:“ űæÔ¡þóè gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚157.405' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.480999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƺ¢;2â- _wandbruntime‚3113  _runtime‚3113c“®Éãà % _timestamp‚1.7884510219020176e+09 _runtime‚3113.456416282 _step‚325 ' lr-AdamW/pg1‚0.00015597132232248202 ' lr-AdamW/pg2‚0.00015597132232248202 trainer/global_step‚3999 çZ‡ÎäÈ _step‚325 ' lr-AdamW/pg1‚0.00015597132232248202 ' lr-AdamW/pg2‚0.00015597132232248202 trainer/global_step‚3999 % _timestamp‚1.7884510219020176e+09 _runtime‚3113.456416282Å={È!2å- _wandbruntime‚3113  _runtime‚3113 )üÛ'æÕO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253782095659520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19163918 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 # _timestamp‚1.78845102190278e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3347673460224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232389925318144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235737598778368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75484501 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _step‚326 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226302726171648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235738273143808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0…<²à'çÚO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3512950415872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335348561234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229638074732882 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3347673460224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232389925318144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235737598778368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75484501 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _step‚326 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226302726171648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235738273143808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253782095659520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33906956 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3347673460224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19163918 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53071329 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 # _timestamp‚1.78845102190278e+09 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 MGDeviceStatsMoMW†Â nitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3114.346879368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3513805408256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884510227917192e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚327 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75482917 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680»fWäì&êæM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58127948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58127946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3114.346879368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3513805408256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884510227917192e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚327 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75482917 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680ÇûË=j2ë- _wandbruntime‚3114  _runtime‚3114qõ5€XìÒ 0loss/FBt3/assignment_loss‚0.8489789563146505 /loss/FBt4/detection_loss‚2.3406471331914265 trainer/global_step‚3999 /loss/FRt2/detection_loss‚1.6680634864538475 0loss/FRt2/assignment_loss‚1.8130686844604602 /loss/FRt3/assignment_loss‚3.469264845762934 /loss/FRt4/assignment_loss‚4.168961348357024 1loss/SRqqt2/assignment_loss‚1.492849903088063 /loss/FBt1/assignment_loss‚0.107987558003515 epoch‚3 .loss/FRt3/detection_loss‚2.233674023832594 % _timestamp‚1.7884510227935984e+09 _step‚328 &loss/total_loss‚48.331890516828864 0loss/FBt4/assignment_loss‚0.5568656499187113 2loss/SRqqt1/assignment_loss‚1.7560936127637707 2loss/SRqqt4/assignment_loss‚2.3458060612384646 .loss/FRt4/detection_loss‚2.914406763182746 1loss/SRqqt2/detection_loss‚1.3264780948559443 .loss/FBt1/detection_loss‚4.253684878349304 /loss/FBt2/detection_loss‚0.7708819408501447 0loss/FRt1/assignment_loss‚3.8292335366660897 0loss/FBt2/assignment_loss‚0.2184410021057511 _runtime‚3114.347928805 2loss/SRqqt3/assignment_loss‚1.6139453648190414 /loss/FRt1/detection_loss‚2.4683931436051023 1loss/SRqqt1/detection_loss‚1.9849097964375517 1loss/SRqqt3/detection_loss‚1.7180041861055153 /loss/FBt3/detection_loss‚2.1982127915729177 0loss/SRqqt4/detection_loss‚2.233037754893303®hIÁ]í× 2loss/SRqqt3/assignment_loss‚1.6139453648190414 /loss/FRt1/detection_loss‚2.4683931436051023 1loss/SRqqt1/detection_loss‚1.9849097964375517 1loss/SRqqt3/detection_loss‚1.7180041861055153 /loss/FBt3/detection_loss‚2.1982127915729177 0loss/SRqqt4/detection_loss‚2.233037754893303 0loss/FBt3/assignment_loss‚0.8489789563146505 /loss/FBt4/detection_loss‚2.3406471331914265 trainer/global_step‚3999 /loss/FRt2/detection_loss‚1.6680634864538475 0loss/FRt2/assignment_loss‚1.8130686844604602 /loss/FRt3/assignment_loss‚3.469264845762934 /loss/FRt4/assignment_loss‚4.168961348357024 1loss/SRqqt2/assignment_loss‚1.492849903088063 /loss/FBt1/assignment_loss‚0.107987558003515 epoch‚3 .loss/FRt3/detection_loss‚2.233674023832594 % _timestamp‚1.7884510227935984e+09 _step‚328 &loss/total_loss‚48.331890516828864 0loss/FBt4/assignment_loss‚0.5568656499187113 2loss/SRqqt1/assignment_loss‚1.7560936127637707 2loss/SRqqt4/assignment_loss‚2.3458060612384646 .loss/FRt4/detection_loss‚2.914406763182746 1loss/SRqqt2/detection_loss‚1.3264780948559443 .loss/FBt1/detection_loss‚4.253684878349304 /loss/FBt2/detection_loss‚0.7708819408501447 0loss/FRt1/assignment_loss‚3.8292335366660897 0loss/FBt2/assignment_loss‚0.2184410021057511 _runtime‚3114.347928805Èo"´ª“î‚(:ˆ Ô±æÔ†€±Õ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.024' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aªûIï‚(:¾ Ô±æÔü×é×% disk./.usageGB‚1110.2233352661133 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd3.in‚0$ memory_percent‚96.59242630004883 disk.sda2.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚74.50801956913399 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 118.95703125" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 proc.cpu.threads‚17 network.recv‚ 1.141851e+06 disk.dm-1.out‚ 118.95703125 disk.rbd0.out‚0& proc.memory.availableMB‚ 558.296875 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 disk.rbd2.in‚0# disk.dm-0.out‚12840.50048828125 disk.sda.in‚ 0.01171875 disk.sda.out‚ 118.95703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625& disk.nvme0n1.out‚12840.50048828125 disk.sda2.out‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1777.out‚0 network.sent‚ 8.715061e+06 disk.rbd2.out‚72.08251953125( disk.nvme0c0n1.out‚12840.50048828125 disk.drbd1777.in‚0 cpu‚25.355789481945283ø„’ð‚(:‡ ã±æÔÞžÁÕ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.639 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý~[@ñ‚(:µ ã±æÔÃë½Ø disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.50801956913399 disk.rbd2.out‚72.09814453125 disk.rbd3.in‚0 disk.sda.out‚ 119.50390625$ memory_percent‚96.59175872802734 network.sent‚ 8.82292e+06 network.recv‚ 1.147541e+06% proc.memory.availableMB‚ 558.40625 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 cpu‚25.35440624591146$ disk.nvme0n1.out‚12840.982421875 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚12840.982421875 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875% disk./.usageGB‚1110.2233352661133! disk.dm-0.out‚12840.982421875 disk.dm-1.out‚ 119.50390625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 119.50390625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875û\‰ä2ò- _wandbruntime‚3149  _runtime‚3149Š|ÇóÁ % _timestamp‚1.7884510575912497e+09 _runtime‚3149.145643552 _step‚329 & lr-AdamW/pg1‚0.0001550053774687976 & lr-AdamW/pg2‚0.0001550053774687976 trainer/global_step‚4049¾—-¤ÌôÆ _step‚329 & lr-AdamW/pg1‚0.0001550053774687976 & lr-AdamW/pg2‚0.0001550053774687976 trainer/global_step‚4049 % _timestamp‚1.7884510575912497e+09 _runtime‚3149.145643552Éãþ2õ- _wandbruntime‚3149  _runtime‚3149¿UÀdö×O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389315628544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58814732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238596910892032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58814730 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34315356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229047949739008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135205245 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚4049 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235207577260544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76410553 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235207595263488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884510575919847e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555908018688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3556075836928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256864656482816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_Ù7ytrain_batch_start/active.large_pool.allocated‚58814732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235207577260544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232424788077406 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235207577260544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76410553 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235207595263488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884510575919847e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555908018688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3556075836928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256864656482816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58814732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235207577260544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256926283280384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76408969 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235263930299392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788451058260312e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53719364 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3377000335606 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _step‚331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76408969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256926307220480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844ÄÜf˜ë&úåM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34323524 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256926307220480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 _runtime‚3149.815193954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235263948302336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256926283280384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76408969 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235263930299392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788451058260312e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53719364 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3377000335606 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _step‚331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76408969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 Bƒ‚(:³ ²æÔ ·­Ø" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.out‚ 121.2734375 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 72.3330078125 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 network.sent‚ 8.944021e+06) disk./.usagePercent‚74.50814014868263 disk.drbd1777.out‚0 disk.dm-0.out‚13038.76171875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 121.2734375 proc.memory.rssMB‚ 2795.90625 disk.dm-1.in‚ 0.01171875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.94140625* proc.memory.percent‚17.064857482910156 proc.cpu.threads‚17 disk.sda.out‚ 121.2734375 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 557.80078125 cpu‚25.350135212041906% disk.nvme0c0n1.out‚13038.76171875 disk.rbd1.out‚ 1.533203125% disk./.usageGB‚1110.2251319885254 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0# disk.nvme0n1.out‚13038.76171875$ memory_percent‚96.59545421600342 network.recv‚ 1.162873e+06 disk.rbd0.in‚0 disk.rbd2.in‚0ï‰b2„- _wandbruntime‚3184  _runtime‚3184û„r]É…Ã % _timestamp‚1.7884510932547162e+09 _runtime‚3184.809085558 _step‚333 ' lr-AdamW/pg1‚0.00015402633202024428 ' lr-AdamW/pg2‚0.00015402633202024428 trainer/global_step‚4099•«eÎ†È _step‚333 ' lr-AdamW/pg1‚0.00015402633202024428 ' lr-AdamW/pg2‚0.00015402633202024428 trainer/global_step‚4099 % _timestamp‚1.7884510932547162e+09 _runtime‚3184.809085558Í›(2‡- _wandbruntime‚3184  _runtime‚3184È Ý'ˆ×O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235210158467698 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59515532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59515530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3598898684416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241455548640256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚3184.809942463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19616998 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54341191 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59515532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77316567 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59515530 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259947193366016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19617433 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263545906583552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59515532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77316567 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59515530 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263546092050432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259947193366016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259947169425920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19617433 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263545906583552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3429645071872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59515530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3598898684416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241455548640256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚3184.809942463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19616998 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54341191 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263608393915392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788451093932502e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3431120875520 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77335021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241512721120256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3185.487373933 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59529546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231848060397568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚335 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3599752042496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3430464499712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417835675014 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238081582241792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263608596146176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414x±¨aë&ŒåM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77335021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241512721120256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3430464499712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3185.487373933 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59529546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231848060397568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚335 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3431120875520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3599752042496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3430464499712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417835675014 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238081582241792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263608596146176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77355085 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263608393915392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788451093932502e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3431120875520 BÈù2—- _wandbruntime‚3220  _runtime‚3220ÐȬxÝ'˜×O trainer/global_step‚4149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78262657 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78242619 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚3220.233682944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54974538 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19842811 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240842899148288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 % _timestamp‚1.7884511286790648e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138458949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78262657 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78242619 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚3220.233682944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54974538 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60216332 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3458509017214 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19842382 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19842811Ò¼ü„‡2š- _wandbruntime‚3220  _runtime‚3220‹‡Öæ&›àM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35140324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642514506752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788451129345515e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4149 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19847360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚339 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234593249886208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642661340160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚3220.900485203 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78281137 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568ÆÈßë&œåM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚3220.900485203 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78281137 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35140324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642514506752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788451129345515e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3472106681344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4149 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 170773504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19847360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚339 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234593249886208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642661340160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344ÓG=î2- _wandbruntime‚3220  _runtime‚3220œþ;]ž× trainer/global_step‚4149 0loss/SRqqt1/detection_loss‚1.678178222133563 /loss/FBt2/detection_loss‚0.5671381210953287 %loss/total_loss‚45.51465624358909 2loss/SRqqt2/assignment_loss‚1.0123816067818552 0loss/SRqqt4/detection_loss‚2.362847282834675 _runtime‚3220.901981723 0loss/FRt1/assignment_loss‚2.7137909176056847 0loss/FRt2/assignment_loss‚2.0558030488061125 /loss/FBt3/detection_loss‚3.1230178938971624 0loss/FBt1/assignment_loss‚0.4139177907782141 1loss/FBt3/assignment_loss‚0.47704028989472946 /loss/FRt2/detection_loss‚1.6195401382006582 1loss/SRqqt3/detection_loss‚1.8161295360209895 .loss/FBt1/detection_loss‚2.728726387023926 % _timestamp‚1.7884511293476217e+09 _step‚340 /loss/FBt4/detection_loss‚2.5883045554161073 epoch‚3 0loss/FRt3/assignment_loss‚3.5347150344597664 2loss/SRqqt1/assignment_loss‚1.4296586931898043 2loss/SRqqt4/assignment_loss‚1.8983008139159367 1loss/FBt2/assignment_loss‚0.21898006452657875 /loss/FRt1/detection_loss‚2.0472313021881536 .loss/FRt3/detection_loss‚2.849142091838937 1loss/SRqqt2/detection_loss‚1.1458469359825054 0loss/FRt4/assignment_loss‚3.8611271381378174 2loss/SRqqt3/assignment_loss‚1.8418860818241156 0loss/FBt4/assignment_loss‚0.5409092873334884 /loss/FRt4/detection_loss‚2.9900430097029758–˱ßbŸÜ /loss/FRt4/detection_loss‚2.9900430097029758 trainer/global_step‚4149 0loss/SRqqt1/detection_loss‚1.678178222133563 /loss/FBt2/detection_loss‚0.5671381210953287 %loss/total_loss‚45.51465624358909 2loss/SRqqt2/assignment_loss‚1.0123816067818552 0loss/SRqqt4/detection_loss‚2.362847282834675 _runtime‚3220.901981723 0loss/FRt1/assignment_loss‚2.7137909176056847 0loss/FRt2/assignment_loss‚2.0558030488061125 /loss/FBt3/detection_loss‚3.1230178938971624 0loss/FBt1/assignment_loss‚0.4139177907782141 1loss/FBt3/assignment_loss‚0.47704028989472946 /loss/FRt2/detection_loss‚1.6195401382006582 1loss/SRqqt3/detection_loss‚1.8161295360209895 .loss/FBt1/detection_loss‚2.728726387023926 % _timestamp‚1.7884511293476217e+09 _step‚340 /loss/FBt4/detection_loss‚2.5883045554161073 epoch‚3 0loss/FRt3/assignment_loss‚3.5347150344597664 2loss/SRqqt1/assignment_loss‚1.4296586931898043 2loss/SRqqt4/assignment_loss‚1.8983008139159367 1loss/FBt2/assignment_loss‚0.21898006452657875 /loss/FRt1/detection_loss‚2.0472313021881536 .loss/FRt3/detection_loss‚2.849142091838937 1loss/SRqqt2/detection_loss‚1.1458469359825054 0loss/FRt4/assignment_loss‚3.8611271381378174 2loss/SRqqt3/assignment_loss‚1.8418860818241156 0loss/FBt4/assignment_loss‚0.5409092873334884Ô»ŸMyž ‚(:“ ½²æÔºî¹Õ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.672' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.734399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÝG^¡‚(:° ½²æÔø“Ø disk.sda2.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 557.67578125 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156) disk./.usagePercent‚74.50820031045316 disk.dm-0.out‚ 13137.765625 disk.dm-1.out‚ 122.75390:ŠœÝ625$ memory_percent‚96.59621715545654 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 122.75390625 network.sent‚ 9.063621e+06 disk.rbd1.in‚0 disk.rbd2.in‚0 cpu‚25.345983765504883 disk.rbd0.out‚0% disk./.usageGB‚1110.2260284423828# disk.nvme0c0n1.out‚ 13137.765625 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd2.out‚ 73.9228515625 network.recv‚ 1.176776e+06" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.out‚ 122.75390625! disk.nvme0n1.out‚ 13137.765625 disk.drbd1777.in‚0„ÂMq’¢‚(:‡ ̲æÔÒëÒÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚148.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.741 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0躅[=£‚(:² ̲æÔÃÌÝÙ# memory_percent‚96.5970516204834% disk./.usageGB‚1110.2261123657227 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 1.182904e+06" disk.nvme0n1.out‚ 13235.3203125 disk.dm-1.out‚ 123.19140625 disk.sda1.in‚0' proc.memory.availableMB‚ 557.5390625 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 network.sent‚ 9.172615e+06 disk.drbd1777.in‚0 cpu‚25.344598680617658 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚74.02880859375 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 13235.3203125 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.dm-0.out‚ 13235.3203125 disk.sda.out‚ 123.19140625 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.out‚0 disk.sda3.out‚ 123.19140625 disk.sda.in‚ 0.01171875) disk./.usagePercent‚74.50820594261891 disk.sda2.out‚0 disk.rbd1.out‚ 1.533203125j]’¤‚(:‡ Û²æÔ±µÈÕ gpu.0.gpu‚35.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡2vWA¥‚(:¶ Û²æÔ¾©Ø disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚74.03662109375" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0$ disk./.usageGB‚1110.225227355957 disk.sda2.in‚0* proc.memory.percent‚17.064857482910156 proc.cpu.threads‚17 network.sent‚ 9.177966e+06) disk./.usagePercent‚74.50814654887098 disk.rbd0.out‚0% disk.nvme0n1.out‚13235.8212890625( proc.memory.availableMB‚ 557.48046875 cpu‚25.343279315730722 disk.sda3.out‚ 123.671875 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625 disk.sda2.out‚0 disk.dm-1.out‚ 123.671875 disk.sda.out‚ 123.671875$ memory_percent‚96.59740924835205 network.recv‚ 1.186673e+06 disk.rbd0.in‚0' disk.nvme0c0n1.out‚13235.8212890625 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0" disk.dm-0.out‚13235.8212890625 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.533203125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0äqgy2¦- _wandbruntime‚3255  _runtime‚3255& S.ȧ ' lr-AdamW/pg1‚0.00015202975815257176 ' lr-AdamW/pg2‚0.00015202975815257176 trainer/global_step‚4199 $ _timestamp‚1.788451164225393e+09 _runtime‚3255.779783283 _step‚341MàXꬂ $ _timestamp‚1.788451164225393e+09 _runtime‚3255.779783283 _step‚341 ' lr-AdamW/pg1‚0.00015202975815257176 ' lr-AdamW/pg2‚0.00015202975815257176 trainer/global_step‚4199ÕÝêÂ2©- _wandbruntime‚3255  _runtime‚3255°¼]:Ý'ª×O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60917132 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20068567 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3255.780708234 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3511616683520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180401152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884511642261307e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240782242202514 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243660533087744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3499344356622 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79188709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414;­ Ââ'«ÜO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60917130 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511616683520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79188709 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884511642261307e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266112243192320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240782242202514 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243660533087744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3499344356622 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79188709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140105841 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60917132 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20068567 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚3255.780708234 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3511616683520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180401152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚343 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79207189 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240837962768038 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60931146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513092487168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512436111360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884511648990765e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243716886126592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60931146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048¢+± ì&®æM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240837962768038 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60931146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513092487168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512436111360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884511648990765e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243716886126592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243716904129536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60931146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3685415026688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚343 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79207189 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808×™/xa2¯- _wandbruntime‚3256  _runtime‚3256ëÉÏY°Ó 0loss/FRt4/assignment_loss‚3.9226283497280545 2loss/SRqqt3/assignment_loss‚2.0358373053992787 1loss/SRqqt1/detection_loss‚1.8365687818399496 0loss/FRt3/assignment_loss‚3.4939508974552154 2loss/SRqqt2/assignment_loss‚1.2680739047795986 /loss/FBt1/detection_loss‚3.0120962162812552 0loss/FBt1/assignment_loss‚0.6226187741073469 1loss/FBt2/assignment_loss‚0.19110442995375065 /loss/FRt3/detection_loss‚2.5103200245648623 /loss/SRqqt4/detection_loss‚2.41734413760049 .loss/FBt3/detection_loss‚2.990083970806815 %loss/total_loss‚46.34837054826599 0loss/FRt2/assignment_loss‚1.3631513576422418 1loss/SRqqt1/assignment_loss‚1.470108237250575 0loss/FBt4/assignment_loss‚0.4812414694748668 % _timestamp‚1.7884511649012158e+09 _step‚344 /loss/FRt2/detection_loss‚1.5968715476670436 .loss/FRt4/detection_loss‚2.917882000958478 1loss/SRqqt2/detection_loss‚0.9153130606494166 /loss/FBt2/detection_loss‚0.5512599319128394 _runtime‚3256.455501319 2loss/SRqqt4/assignment_loss‚1.9046799323388508 /loss/FBt3/assignment_loss‚0.404169469388266 .loss/FRt1/detection_loss‚2.426387129945958 epoch‚3 1loss/SRqqt3/detection_loss‚1.9362462839732568 /loss/FBt4/detection_loss‚3.0399221728245416 trainer/global_step‚4199 0loss/FRt1/assignment_loss‚3.0405111617230354X”^±Ø 2loss/SRqqt4/assignment_loss‚1.9046799323388508 /loss/FBt3/assignment_loss‚0.404169469388266 .loss/FRt1/detection_loss‚2.426387129945958 epoch‚3 1loss/SRqqt3/detection_loss‚1.9362462839732568 /loss/FBt4/detection_loss‚3.0399221728245416 trainer/global_step‚4199 0loss/FRt1/assignment_loss‚3.0405111617230354 0loss/FRt4/assignment_loss‚3.9226283497280545 2loss/SRqqt3/assignment_loss‚2.0358373053992787 1loss/SRqqt1/detection_loss‚1.8365687818399496 0loss/FRt3/assignment_loss‚3.4939508974552154 2loss/SRqqt2/assignment_loss‚1.2680739047795986 /loss/FBt1/detection_loss‚3.0120962162812552 0loss/FBt1/assignment_loss‚0.6226187741073469 1loss/FBt2/assignment_loss‚0.19110442995375065 /loss/FRt3/detection_loss‚2.5103200245648623 /loss/SRqqt4/detection_loss‚2.41734413760049 .loss/FBt3/detection_loss‚2.990083970806815 %loss/total_loss‚46.34837054826599 0loss/FRt2/assignment_loss‚1.3631513576422418 1loss/SRqqt1/assignment_loss‚1.470108237250575 0loss/FBt4/assignment_loss‚0.4812414694748668 % _timestamp‚1.7884511649012158e+09 _step‚344 /loss/FRt2/detection_loss‚1.5968715476670436 .loss/FRt4/detection_loss‚2.917882000958478 1loss/SRqqt2/detection_loss‚0.9153130606494166 /loss/FBt2/detection_loss‚0.5512599319128394 _runtime‚3256.455501319ØængÇ“²‚(:ˆ ê²æÔÿÜÞÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³éÆD³‚(:¹ ê²æﻃۯ network.sent‚ 9.184089e+06" disk.dm-0.out‚13333.3095703125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 124.1015625* proc.memory.percent‚17.064857482910156 disk.dm-0.in‚ 162.94140625 disk.rbd2.in‚0$ memory_percent‚96.59810066223145 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17) disk./.usagePercent‚74.50821413486001 disk.drbd1777.out‚0 disk.sda.out‚ 124.1015625" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875' proc.memory.availableMB‚ 557.3671875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 1.533203125 disk.rbd3.out‚0 disk.rbd2.out‚75.62646484375 network.recv‚ 1.190843e+06' disk.nvme0c0n1.out‚13333.3095703125 disk.sda1.out‚0% disk./.usageGB‚1110.2262344360352% disk.nvme0n1.out‚13333.3095703125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 124.1015625 cpu‚25.341944496345455‘+¶k“´‚(:ˆ ù²æÔÒ½ëÕ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S|O·Gµ‚(:¼ ù²æÔ±‰ÕØ) disk./.usagePercent‚74.50821413486001 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 124.6953125( disk.nvme0c0n1.out‚13430.90673828125 disk.sda2.out‚0* proc.memory.percent‚17.064857482910156% disk./.usageGB‚1110.2262344360352 disk.sda1.in‚0 disk.rbd1.in‚0 cpu‚25.34061118804476 network.sent‚ 9.292454e+06 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.533203125 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 124.6953125 disk.rbd2.out‚75.62646484375& disk.nvme0n1.out‚13430.90673828125 disk.dm-1.out‚ 124.6953125 disk.rbd2.in‚0 network.recv‚ 1.196865e+06$ memory_percent‚96.59812450408936( proc.memory.availableMB‚ 557.36328125 disk.nvme0n1.in‚ 162.94140625 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚13430.90673828125 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625Àæ32¶- _wandbruntime‚3291  _runtime‚3291v¹·(È·Â _runtime‚3291.241417228 _step‚345 ' lr-AdamW/pg1‚0.00015101264581645406 ' lr-AdamW/pg2‚0.00015101264581645406 trainer/global_step‚4249 $ _timestamp‚1.788451199686992e+09)#Ü#Í¸Ç ' lr-AdamW/pg1‚0.00015101264581645406 ' lr-AdamW/pg2‚0.00015101264581645406 trainer/global_step‚4249 $ _timestamp‚1.788451199686992e+09 _runtime‚3291.241417228 _step‚345Ùýlkˆ2¹- _wandbruntime‚3291  _runtime‚3291ÿäcÃÜ'ºÖO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80094723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250031461884928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3540179696030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250031461884928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚4249Sau‘»ÛO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20294668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3540179696030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250031461884928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3553258851840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3552602489344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243568955547038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚4249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚246478185030144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80094723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250031461884928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250030787519488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932ÚaõÀW2¼- _wandbruntime‚3291  _runtime‚3291¹3Uç&½áM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553421917184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20298589 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243624004635446 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272984726304768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272984532462592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250088634364928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250088634364928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35957124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3728271551488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚347 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269256430813184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0.nM’ì&¾æM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚347 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269256430813184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3553421917184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20298589 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243624004635446 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272984726304768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246534556071936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272984532462592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250088634364928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80133241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250088634364928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35957124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3553421917184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3728271551488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540996131202Û‚Y2¿- _wandbruntime‚3291  _runtime‚3291ol!aUÀÏ -loss/FRt1/detection_loss‚2.43111927769123 .loss/FRt2/detection_loss‚1.557114644907415 /loss/FRt3/detection_loss‚2.0491441739723086 /loss/FRt3/assignment_loss‚3.257757837884128 0loss/FRt4/assignment_loss‚4.4401303112506865 1loss/FBt1/assignment_loss‚0.46925425715744495 0loss/SRqqt4/detection_loss‚3.141041089187969 0loss/SRqqt1/detection_loss‚2.202252036333084 epoch‚3 2loss/SRqqt1/assignment_loss‚1.2131483756937087 1loss/SRqqt3/detection_loss‚2.0380084171544675 % _timestamp‚1.7884512003623416e+09 2loss/SRqqt4/assignment_loss‚1.8703738288445906 .loss/FRt4/detection_loss‚3.480233830213547 1loss/SRqqt2/detection_loss‚1.3276879806603705 .loss/FBt1/detection_loss‚2.912377320230007 /loss/FBt4/detection_loss‚2.8712839484214783 %loss/total_loss‚47.11485769529493 .loss/FRt1/assignment_loss‚3.10996585778701 0loss/FBt3/assignment_loss‚0.3024242426141758 /loss/FBt2/detection_loss‚0.6955932585690927 trainer/global_step‚4249 _runtime‚3291.916917583 0loss/FRt2/assignment_loss‚1.7190854493528605 /loss/FBt4/assignment_loss‚0.800662565444197 /loss/FBt3/detection_loss‚2.2112758701497857 2loss/SRqqt2/assignment_loss‚1.1357091573333102 2loss/SRqqt3/assignment_loss‚1.6538463169059088 0loss/FBt2/assignment_loss‚0.2253676475361593 _step‚348„º“UZÁÔ /loss/FRt3/assignment_loss‚3.257757837884128 0loss/FRt4/assignment_loss‚4.4401303112506865 1loss/FBt1/assignment_loss‚0.46925425715744495 0loss/SRqqt4/detection_loss‚3.141041089187969 0loss/SRqqt1/detection_loss‚2.202252036333084 epoch‚3 2loss/SRqqt1/assignment_loss‚1.2131483756937087 1loss/SRqqt3/detection_loss‚2.0380084171544675 % _timestamp‚1.7884512003623416e+09 2loss/SRqqt4/assignment_loss‚1.8703738288445906 .loss/FRt4/detection_loss‚3.480233830213547 1loss/SRqqt2/detection_loss‚1.3276879806603705 .loss/FBt1/detection_loss‚2.912377320230007 /loss/FBt4/detection_loss‚2.8712839484214783 %loss/total_loss‚47.11485769529493 .loss/FRt1/assignment_loss‚3.10996585778701 0loss/FBt3/assignment_loss‚0.3024242426141758 /loss/FBt2/detection_loss‚0.6955932585690927 trainer/global_step‚4249 _runtime‚3291.916917583 0loss/FRt2/assignment_loss‚1.7190854493528605 /loss/FBt4/assignment_loss‚0.800662565444197 /loss/FBt3/detection_loss‚2.2112758701497857 2loss/SRqqt2/assignment_loss‚1.1357091573333102 2loss/SRqqt3/assignment_loss‚1.6538463169059088 0loss/FBt2/assignment_loss‚0.2253676475361593 _step‚348 -loss/FRt1/detection_loss‚2.43111927769123 .loss/FRt2/detection_loss‚1.557114644907415 /loss/FRt3/detection_loss‚2.0491441739723086ÜÐ&¦l’‚(:‡ ˆ³æÔÀàèÖ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²V ¨JÂ(:¿ ˆ³æÔï¡žÚ disk.rbd1.in‚0 disk.rbd3.out‚0 cpu‚25.33927563257188 network.recv‚ 1.201263e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0% disk./.usageGB‚1110.2262344360352 disk.sda3.out‚ 125.14453125 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚13431.28173828125 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 disk.sda.out‚ 125.14453125 network.sent‚ 9.298689e+06" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0n1.out‚13431.28173828125 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0$ memory_percent‚96.59984111785889 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚13431.28173828125 disk.drbd1777.in‚0( proc.memory.availableMB‚ 557.08203125* proc.memory.percent‚17.064857482910156) disk./.usagePercent‚74.50821413486001 disk.rbd2.out‚75.72021484375 disk.drbd1777.out‚0 disk.dm-1.out‚ 125.14453125Ë<ܞĂ(:“ —³æÔ˜²ËÕ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚101.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.227199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÇØNDÅ‚(:¹ —³æÔÜ·‹Ù disk.sda.out‚ 125.8359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚75.72021484375% disk./.usageGB‚1110.2262535095215 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625 cpu‚25.337994887543598 proc.cpu.threads‚17) disk./.usagePercent‚74.50821541489769' disk.nvme0c0n1.out‚13529.0595703125 disk.dm-1.out‚ 125.8359375 disk.sda3.out‚ 125.8359375 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚13529.0595703125 disk.rbd1.out‚ 1.533203125( proc.memory.availableMB‚ 557.08203125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875% disk.nvme0n1.out‚13529.0595703125 disk.rbd1.in‚0* proc.memory.percent‚17.064857482910156 network.recv‚ 1.20751e+06 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd3.in‚0$ memory_percent‚96.59984111785889 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 network.sent‚ 9.406834e+06 disk.sda2.in‚0 disk.rbd3.out‚0 ½‰ˆ2Æ- _wandbruntime‚3327  _runtime‚3327xÙÉÇà _runtime‚3327.980389294 _step‚349 ' lr-AdamW/pg1‚0.00014998326496475938 ' lr-AdamW/pg2‚0.00014998326496475938 trainer/global_step‚4299 % _timestamp‚1.7884512364259434e+09'£ÎÈÈ _runtime‚3327.980389294 _step‚349 ' lr-AdamW/pg1‚0.00014998326496475938 ' lr-AdamW/pg2‚0.00014998326496475938 trainer/global_step‚4299 % _timestamp‚1.7884512364259434e+09ÝE32É- _wandbruntime‚3327  _runtime‚3327ଡÊÖO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 G>DeviceStatsMonitor.on_trë¿õ;&ain_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272277340898816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249295836972544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81020775 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252889425267712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚4299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890099633152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2µäÌá'ËÛO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚4299 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890099633152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272277340898816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249295836972544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81020775 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252889425267712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Þiå2Ì- _wandbruntime‚3328  _runtime‚3328ÆY(Fç&ÍáM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56890015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272338967696384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252947272113152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242828886509568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884512371153295e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚4299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276110047450624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚3328.670177219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20524489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246410717980178 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81039229 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844)Ó<¨ ÎæM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249352208014336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20524489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246410717980178 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81039229 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56890015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20524044 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272338967696384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252947272113152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242828886509568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884512371153295e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚4299 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276110047450624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚3328.670177219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3635230463488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252113488914944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255748063015936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245519188907008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚3364.544387417 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245519171867648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872Z¦éËÝÜO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚245519188907008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚3364.544387417 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63019530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81946827 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245519171867648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252113488914944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3635230463488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252113488914944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255748063015936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20745906âÜÚ„ã2Þ- _wandbruntime‚3365  _runtime‚3365Óù,›ç&ßáM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245574075998208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252169841953792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63033548 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚355 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63033546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252169859956736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3814141783040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81985345 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20750350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63033548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3365.241595045 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884512736867156e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144998827 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 trainer/global_step‚4349 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752ˆ­æ×ì&àæM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36773926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144998827 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 trainer/global_step‚4349 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81965281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245574075998208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252169841953792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63033548 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275421528519680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚355 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63033546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635393528832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252169859956736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3814141783040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81985345 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20750350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63033548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255805909861376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3365.241595045 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884512736867156e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088ãÿ¸*2á- _wandbruntime‚3365  _runtime‚33650ÑåaâÛ 1loss/FBt3/assignment_loss‚0.25542160868280916 /loss/FRt4/detection_loss‚3.2708482515244257 trainer/global_step‚4349 $ _timestamp‚1.788451273688573e+09 /loss/FRt2/detection_loss‚1.7651652423871889 /loss/FRt3/detection_loss‚2.1750372173609556 1loss/SRqqt4/detection_loss‚2.7507782044081854 /loss/FBt3/detection_loss‚2.3576521147042513 /loss/FBt4/detection_loss‚3.4648647563798085 _runtime‚3365.242876259 1loss/FBt2/assignment_loss‚0.25434610396089075 /loss/FBt2/detection_loss‚0.7376986390455367 0loss/FRt3/assignment_loss‚3.1697090257097176 1loss/SRqqt3/assignment_loss‚1.572584192356411 1loss/FBt4/assignment_loss‚0.19756688416314105 1loss/SRqqt1/detection_loss‚1.7872740372698357 2loss/SRqqt4/assignment_loss‚2.1408689742715192 /loss/FRt1/detection_loss‚2.2411221186725463 2loss/SRqqt2/assignment_loss‚0.8836517162562814 1loss/FBt1/assignment_loss‚0.20735399276014732 1loss/SRqqt2/detection_loss‚1.1890153011772782 _step‚356 0loss/FRt1/assignment_loss‚2.8416308115939706 0loss/FRt2/assignment_loss‚1.3593065566486782 1loss/SRqqt1/assignment_loss‚1.099531213198095 1loss/SRqqt3/detection_loss‚1.6430915405727782 /loss/FBt1/detection_loss‚3.8988598346710206 &loss/total_loss‚46.176263714160754 0loss/FRt4/assignment_loss‚4.9128853763852804 epoch‚3æìÄfãà /loss/FRt1/detection_loss‚2.2411221186725463 2loss/SRqqt2/assignment_loss‚0.8836517162562814 1loss/FBt1/assignment_loss‚0.20735399276014732 1loss/SRqqt2/detection_loss‚1.1890153011772782 _step‚356 0loss/FRt1/assignment_loss‚2.8416308115939706 0loss/FRt2/assignment_loss‚1.3593065566486782 1loss/SRqqt1/assignment_loss‚1.099531213198095 1loss/SRqqt3/detection_loss‚1.6430915405727782 /loss/FBt1/detection_loss‚3.8988598346710206 &loss/total_loss‚46.176263714160754 0loss/FRt4/assignment_loss‚4.9128853763852804 epoch‚3 1loss/FBt3/assignment_loss‚0.25542160868280916 /loss/FRt4/detection_loss‚3.2708482515244257 trainer/global_step‚4349 $ _timestamp‚1.788451273688573e+09 /loss/FRt2/detection_loss‚1.7651652423871889 /loss/FRt3/detection_loss‚2.1750372173609556 1loss/SRqqt4/detection_loss‚2.7507782044081854 /loss/FBt3/detection_loss‚2.3576521147042513 /loss/FBt4/detection_loss‚3.4648647563798085 _runtime‚3365.242876259 1loss/FBt2/assignment_loss‚0.25434610396089075 /loss/FBt2/detection_loss‚0.7376986390455367 0loss/FRt3/assignment_loss‚3.1697090257097176 1loss/SRqqt3/assignment_loss‚1.572584192356411 1loss/FBt4/assignment_loss‚0.19756688416314105 1loss/SRqqt1/detection_loss‚1.7872740372698357 2loss/SRqqt4/assignment_loss‚2.1408689742715192äП¶~œä‚(:‘ Ó³æÔ­ÞÕ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r±Ù<å‚(:± Ó³æÔƒÖâÙ proc.memory.rssMB‚ 2795.90625 cpu‚25.332903042901446 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚13725.2255859375% disk.nvme0n1.out‚13725.2255859375 disk.rbd1.out‚ 1.681640625) disk./.usagePercent‚74.50817010156413 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.out‚ 127.90625 disk.rbd3.in‚0$ memory_percent‚96.60050868988037 network.sent‚ 9.53289e+06 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 77.482421875 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 127.90625 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875% disk./.usageGB‚1110.2255783081055 disk.sda.out‚ 127.90625 disk.sda1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 556.97265625* proc.memory.percent‚17.064857482910156 disk.rbd2.in‚0" disk.dm-0.out‚13725.2255859375 proc.cpu.threads‚17 network.recv‚ 1.225818e+06 disk.rbd0.out‚0 disk.sda2.out‚0\Øã“æ‚(:ˆ â³æÔÅȾÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç;£jHç‚(:½ â³æÔ–›Ù disk.rbd1.out‚ 1.681640625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 128.45703125 disk.rbd2.in‚0' disk.nvme0c0n1.out‚13725.4833984375 network.sent‚ 9.641843e+06 network.recv‚ 1.231915e+06% disk.nvme0n1.out‚13725.4833984375 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875% disk./.usageGB‚1110.2255783081055 disk.rbd1.in‚0 disk.rbd2.out‚77.58447265625 disk.sda2.out‚0* proc.memory.percent‚17.064857482910156 cpu‚25.331669258877117 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 557.21484375 proc.memory.rssMB‚ 2795.90625) disk./.usagePercent‚74.50817010156413" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 162.94140625" disk.dm-0.out‚13725.4833984375 disk.dm-1.out‚ 128.45703125 disk.sda.out‚ 128.45703125$ memory_percent‚96.599030494689947óv2è- _wandbruntime‚3401  _runtime‚3401g ÌyÃé½ $ lr-AdamW/pg1‚0.00014788855831238 $ lr-AdamW/pg2‚0.00014788855831238 trainer/global_step‚4399 % _timestamp‚1.7884513097418797e+09 _runtime‚3401.296312813 _step‚357­-JÈê _runtime‚3401.296312813 _step‚357 $ lr-AdamW/pg1‚0.00014788855831238 $ lr-AdamW/pg2‚0.00014788855831238 trainer/global_step‚4399 % _timestamp‚1.7884513097418797e+09åC„ïÇ2ë- _wandbruntime‚3401  _runtime‚3401ÚFxW&ì×O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884513097426445e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282298637216768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58147052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254931140857344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146613249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254931158860288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63720330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258606700764160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972894 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278442414665216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282298441264128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3856050539008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844›PÑâ'íÜO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248264378395648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146613249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254931158860288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63720330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258606700764160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972894 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3676216269312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278442414665216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282298441264128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3856050539008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884513097426445e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63720330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282298637216768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82872879 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58147052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63720332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254931140857344æßD2î- _wandbruntime‚3401  _runtime‚3401Iíh+ç&ïáM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37182324 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4399 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82891333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚258664547609600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146645745 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3677035710464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248319299565568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20977414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 _runtime‚3401.980461945 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58159740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚359 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146625679 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278504041462784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20976979•˜B+ðæM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 _runtime‚3401.980461945 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58159740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚359 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146625679 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663502149426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278504041462784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20976979 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37182324 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4399 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82891333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚258664547609600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63734348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGD†Š'ÁeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146645745 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3677035710464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248319299565568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20977414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0çGmv;2ñ- _wandbruntime‚3401  _runtime‚3401 ±ÏYòÓ 1loss/SRqqt2/detection_loss‚1.0087847388893754 .loss/FBt1/detection_loss‚2.049173847834269 epoch‚3 _runtime‚3401.981349167 2loss/SRqqt1/assignment_loss‚1.6859901461750268 .loss/FRt1/detection_loss‚2.721979782348726 1loss/SRqqt3/detection_loss‚1.6216059152096036 % _timestamp‚1.7884513104270356e+09 0loss/FRt3/assignment_loss‚3.2216181646693838 /loss/FRt4/assignment_loss‚4.508470307225767 2loss/SRqqt2/assignment_loss‚1.3064762943654091 0loss/FBt3/assignment_loss‚1.1745066709816456 /loss/FRt3/detection_loss‚2.1500567081299695 0loss/SRqqt4/detection_loss‚2.052962197611729 0loss/FRt1/assignment_loss‚3.0759923312722184 2loss/SRqqt3/assignment_loss‚1.2500231461155982 0loss/FBt1/assignment_loss‚0.6571437967320283 /loss/FBt2/detection_loss‚0.8557058460077497 _step‚360 /loss/FRt2/assignment_loss‚1.980936096886457 1loss/FBt2/assignment_loss‚0.27110467155085427 1loss/SRqqt1/detection_loss‚1.7490725354714827 .loss/FBt4/detection_loss‚3.664619117975235 1loss/FBt4/assignment_loss‚0.12934019800732344 .loss/FBt3/detection_loss‚3.213097262382507 2loss/SRqqt4/assignment_loss‚2.1366290282458067 /loss/FRt2/detection_loss‚1.9760152492987908 .loss/FRt4/detection_loss‚3.144555874492811 %loss/total_loss‚47.60585992787976 trainer/global_step‚4399NcÐ^óØ 2loss/SRqqt1/assignment_loss‚1.6859901461750268 .loss/FRt1/detection_loss‚2.721979782348726 1loss/SRqqt3/detection_loss‚1.6216059152096036 % _timestamp‚1.7884513104270356e+09 0loss/FRt3/assignment_loss‚3.2216181646693838 /loss/FRt4/assignment_loss‚4.508470307225767 2loss/SRqqt2/assignment_loss‚1.3064762943654091 0loss/FBt3/assignment_loss‚1.1745066709816456 /loss/FRt3/detection_loss‚2.1500567081299695 0loss/SRqqt4/detection_loss‚2.052962197611729 0loss/FRt1/assignment_loss‚3.0759923312722184 2loss/SRqqt3/assignment_loss‚1.2500231461155982 0loss/FBt1/assignment_loss‚0.6571437967320283 /loss/FBt2/detection_loss‚0.8557058460077497 _step‚360 /loss/FRt2/assignment_loss‚1.980936096886457 1loss/FBt2/assignment_loss‚0.27110467155085427 1loss/SRqqt1/detection_loss‚1.7490725354714827 .loss/FBt4/detection_loss‚3.664619117975235 1loss/FBt4/assignment_loss‚0.12934019800732344 .loss/FBt3/detection_loss‚3.213097262382507 2loss/SRqqt4/assignment_loss‚2.1366290282458067 /loss/FRt2/detection_loss‚1.9760152492987908 .loss/FRt4/detection_loss‚3.144555874492811 %loss/total_loss‚47.60585992787976 trainer/global_step‚4399 1loss/SRqqt2/detection_loss‚1.0087847388893754 .loss/FBt1/detection_loss‚2.049173847834269 epoch‚3 _runtime‚3401.981349167èn•Ôžô‚(:“ ñ³æÔœöüÕ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.187' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.837400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·.OHõ‚(:½ ñ³æÔëùÞØ disk.rbd3.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚96.60053253173828 disk.rbd2.in‚0 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 network.sent‚ 9.646553e+06 network.recv‚ 1.235344e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚13823.24462890625 disk.drbd1777.in‚0 cpu‚25.330444738477535 disk.rbd1.out‚ 1.7138671875 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚13823.24462890625 disk.nvme0n1.in‚ 162.94140625 disk.sda1.in‚0 disk.rbd2.out‚77.58447265625 disk.drbd1777.out‚0# disk.dm-0.out‚13823.24462890625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.dm-1.out‚ 128.91796875 disk.sda.out‚ 128.91796875 disk.dm-0.in‚ 162.94140625% proc.memory.availableMB‚ 556.96875 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 128.91796875 disk.sda2.out‚0( disk./.usagePercent‚74.5082356394929% disk./.usageGB‚1110.2265548706055–!Ö±žö‚(:“ €´æÔ´½÷Õ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.517599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dz™ÝH÷‚(:½ €´æÔâ·Ù disk.dm-0.in‚ 162.94140625 disk.sda1.in‚0% disk./.usageGB‚1110.2265548706055 disk.rbd3.out‚0 disk.sda.out‚ 129.3359375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚77.66259765625 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.dm-1.out‚ 129.3359375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 network.recv‚ 1.241811e+06( disk./.usagePercent‚74.5082356394929 disk.rbd1.out‚ 1.7138671875" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625* proc.memory.percent‚17.064857482910156 disk.rbd2.in‚0& disk.nvme0n1.out‚13920.76416015625 disk.drbd1777.out‚0( proc.memory.availableMB‚ 556.66015625 network.sent‚ 9.755453e+06( disk.nvme0c0n1.out‚13920.76416015625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚96.60241603851318 cpu‚25.329286070981762 disk.sda3.out‚ 129.3359375# disk.dm-0.out‚13920.76416015625 proc.memory.rssMB‚ 2795.90625 proc.cpu.threads‚17ÆcH“ø‚(:ˆ ´æÔ÷΂Ö gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÅÕLù‚(:Á ´æÔþ—»Ø disk.rbd0.in‚0 disk.rbd2.out‚79.17431640625 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.77685546875( proc.memory.availableMB‚ 557.12109375 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156 network.recv‚ 1.245223e+06 disk.sda.out‚ 130.04296875 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.328090090866915 network.sent‚ 9.760685e+06% disk./.usageGB‚1110.2265548706055 disk.rbd3.out‚0$ memory_percent‚96.59960269927979( disk./.usagePercent‚74.5082356394929 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚13921.35107421875 disk.nvme0n1.in‚ 162.94140625& disk.nvme0n1.out‚13921.35107421875 proc.cpu.threads‚17 disk.sda3.out‚ 130.04296875 disk.rbd2.in‚0 disk.dm-0.in‚ 162.94140625 disk.dm-1.out‚ 130.04296875 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚13921.35107421875 disk.sda2.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.94140625íž{í2ú- _wandbruntime‚3437  _runtime‚3437l…ÂÄû¾ _runtime‚3437.867969905 _step‚361 % lr-AdamW/pg1‚0.000146823669045422 % lr-AdamW/pg2‚0.000146823669045422 trainer/global_step‚4449 $ _timestamp‚1.788451346313593e+09z¶]¦Éüà % lr-AdamW/pg1‚0.000146823669045422 % lr-AdamW/pg2‚0.000146823669045422 trainer/global_step‚4449 $ _timestamp‚1.788451346313593e+09 _runtime‚3437.867969905 _step‚361é_÷2ý- _wandbruntime‚3437  _runtime‚3437(x#õÝ'þ×O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281524927608320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4449 % _timestamp‚1.7884513463143022e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3703521053662 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257748810802688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257748792799744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37582556 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251009584923648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83818969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3899096997376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281524927608320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4449 % _timestamp‚1.7884513463143022e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3703521053662 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257748810802688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257748792799744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37582556 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251009584923648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83818969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3899096997376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 J8DeviceStôQ~™%atsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3704337488834 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3703683050870 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254768843582402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21204546 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83817385 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58795272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257805145838592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257805163841536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257805163841536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 " _timestamp‚1.7884513469981e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3717365140480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3718021516288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522510979072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148252531 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285486724358144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3438.553407698 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3718021516288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261522510979072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148252531 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261523185357824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281586602286080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285486724358144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3438.553407698 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3704337488834 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3703683050870 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254768843582402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21204546 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83817385 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58795272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64435148 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257805145838592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257805163841536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899960558592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254768172105078 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83837449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83817385ëÅë2ƒ- _wandbruntime‚3438  _runtime‚3438Bå3sW„Ñ 0loss/SRqqt1/detection_loss‚2.033799958161332 /loss/FBt3/detection_loss‚2.5059911757707596 trainer/global_step‚4449 _step‚364 1loss/FBt4/assignment_loss‚0.28378312019534135 0loss/FBt3/assignment_loss‚0.4728690224001184 0loss/SRqqt4/detection_loss‚2.546126083020241 .loss/FBt1/detection_loss‚2.632070541381836 /loss/FBt2/detection_loss‚0.6153945258107526 2loss/SRqqt1/assignment_loss‚1.6161255795847287 2loss/SRqqt2/assignment_loss‚1.1088752072161208 % _timestamp‚1.7884513470003886e+09 _runtime‚3438.554691086 0loss/FRt2/assignment_loss‚1.6136938040373756 0loss/FBt2/assignment_loss‚0.1668295636274677 /loss/FRt2/detection_loss‚1.6131245972680264 1loss/SRqqt3/detection_loss‚1.9619284713709797 /loss/FBt4/detection_loss‚2.8937887847423553 /loss/FRt1/detection_loss‚2.4201953300209933 0loss/FBt1/assignment_loss‚0.3323142644284027 epoch‚3 1loss/SRqqt4/assignment_loss‚1.822363342008283 .loss/FRt3/detection_loss‚2.226800559007603 /loss/FRt4/detection_loss‚3.1645291867079557 %loss/total_loss‚44.76514496247721 0loss/FRt1/assignment_loss‚2.7853733467501263 /loss/FRt3/assignment_loss‚3.094902366399765 /loss/FRt4/assignment_loss‚4.186132024835657 2loss/SRqqt3/assignment_loss‚1.4381734554621357 /loss/SRqqt2/detection_loss‚1.229960652268850º\…Ö /loss/FRt4/detection_loss‚3.1645291867079557 %loss/total_loss‚44.76514496247721 0loss/FRt1/assignment_loss‚2.7853733467501263 /loss/FRt3/assignment_loss‚3.094902366399765 /loss/FRt4/assignment_loss‚4.186132024835657 2loss/SRqqt3/assignment_loss‚1.4381734554621357 /loss/SRqqt2/detection_loss‚1.22996065226885 0loss/SRqqt1/detection_loss‚2.033799958161332 /loss/FBt3/detection_loss‚2.5059911757707596 trainer/global_step‚4449 _step‚364 1loss/FBt4/assignment_loss‚0.28378312019534135 0loss/FBt3/assignment_loss‚0.4728690224001184 0loss/SRqqt4/detection_loss‚2.546126083020241 .loss/FBt1/detection_loss‚2.632070541381836 /loss/FBt2/detection_loss‚0.6153945258107526 2loss/SRqqt1/assignment_loss‚1.6161255795847287 2loss/SRqqt2/assignment_loss‚1.1088752072161208 % _timestamp‚1.7884513470003886e+09 _runtime‚3438.554691086 0loss/FRt2/assignment_loss‚1.6136938040373756 0loss/FBt2/assignment_loss‚0.1668295636274677 /loss/FRt2/detection_loss‚1.6131245972680264 1loss/SRqqt3/detection_loss‚1.9619284713709797 /loss/FBt4/detection_loss‚2.8937887847423553 /loss/FRt1/detection_loss‚2.4201953300209933 0loss/FBt1/assignment_loss‚0.3323142644284027 epoch‚3 1loss/SRqqt4/assignment_loss‚1.822363342008283 .loss/FRt3/detection_loss‚2.226800559007603ì):t…ž†‚(:“ ž´æÔ²¤ÍÕ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.635' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.726999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ;M|J‡‚(:¿ ž´æÔŸÈ™Ø disk.drbd1777.in‚0 disk.rbd0.in‚0 network.sent‚ 9.766876e+06 network.recv‚ 1.249439e+06% disk./.usageGB‚1110.2265586853027( disk.nvme0c0n1.out‚14018.81201171875 disk.sda2.out‚0 disk.rbd3.in‚0 proc.memory.rssMB‚ 2795.90625 disk.rbd2.out‚79.25244140625 disk.dm-1.in‚ 0.01171875* proc.memory.percent‚17.064857482910156 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 130.59765625# memory_percent‚96.6010570526123 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625# disk.dm-0.out‚14018.81201171875 disk.sda2.in‚0 disk.dm-1.out‚ 130.59765625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0' proc.memory.availableMB‚ 556.8828125 cpu‚25.32682206531705) disk./.usagePercent‚74.50823589550043 disk.sda.out‚ 130.59765625& disk.nvme0n1.out‚14018.81201171875 disk.rbd1.out‚ 1.77685546875" disk.nvme0c0n1.in‚ 162.94140625Pe“ˆ‚(:ˆ ­´æÔÈŠÁÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v÷žC‰‚(:¸ ­´æÔÝ××Ø disk.drbd1777.out‚0) disk./.usagePercent‚74.50823589550043 disk.rbd3.out‚0 proc.memory.rssMB‚ 2795.90625 disk.rbd2.out‚79.25244140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.064857482910156 disk.rbd2.in‚0' proc.memory.availableMB‚ 556.8515625 cpu‚25.325696938463572 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 131.1328125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 network.sent‚ 9.875138e+06% disk./.usageGB‚1110.2265586853027 disk.rbd3.in‚0 disk.sda.out‚ 131.1328125 disk.nvme0n1.in‚ 162.94140625 disk.sda1.in‚0' disk.nvme0c0n1.out‚14019.4150390625% disk.nvme0n1.out‚14019.4150390625 network.recv‚ 1.255668e+06" disk.dm-0.out‚14019.4150390625 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0 disk.dm-1.out‚ 131.1328125$ memory_percent‚96.60124778747559 proc.cpu.threads‚17 disk.drbd1777.in‚0·oA–2Š- _wandbruntime‚3474  _runtime‚3474B\¸\É‹Ã _runtime‚3474.611228025 _step‚365 ' lr-AdamW/pg1‚0.00014574738423936756 ' lr-AdamW/pg2‚0.00014574738423936756 trainer/global_step‚4499 % _timestamp‚1.7884513830567937e+09ÄÏyÎŒÈ ' lr-AdamW/pg2‚0.00014574738423936756 trainer/global_step‚4499 % _timestamp‚1.7884513830567937e+09 _runtime‚3474.611228025 _step‚365 ' lr-AdamW/pg1‚0.00014574738423936756íL®+Œ2- _wandbruntime‚3474  _runtime‚3474s·‹–~Ž×O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3743701955314 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37990958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257498493406962 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758187880960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65121930 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65121932 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260566444742144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284607488431616Š×„sâ'ÜO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59416176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253754808491008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21425220 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253754791451648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 % _timestamp‚1.7884513830576963e+09 _runtime‚3474.612389954 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288549721772544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3744356393070 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65121932 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260566444742144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284607488431616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3743701955314 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37990958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257498493406962 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758187880960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65121930 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3942921348608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264381148727296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257554213972486 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264381823106048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21430136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758350946304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3744518390278 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3475.312671041 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65135946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65135948 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3745172828242 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260622815783936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84763501 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844y©Ê”…#’äM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3745172828242 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260622815783936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84763501 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3942921348608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264381148727296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21429680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257554213972486 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264381823106048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21430136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758350946304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3744518390278 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3475.312671041 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879383 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65135946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65135948 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304ïú!22“- _wandbruntime‚3475  _runtime‚3475GI†û[”Õ /loss/FBt4/detection_loss‚2.8994919300079345 2loss/SRqqt4/assignment_loss‚1.9603838950395585 1loss/FBt3/assignment_loss‚0.36427606602332424 _step‚368 0loss/SRqqt4/detection_loss‚2.454655526081721 /loss/FBt2/detection_loss‚0.6658392310673664 /loss/FBt3/detection_loss‚2.9753264018467496 trainer/global_step‚4499 1loss/SRqqt2/assignment_loss‚1.006911070406204 .loss/FRt4/detection_loss‚3.514742116133372 1loss/SRqqt2/detection_loss‚1.2502818417269737 epoch‚3 0loss/FRt4/assignment_loss‚3.6294554670651755 1loss/SRqqt1/assignment_loss‚1.496059173293281 1loss/SRqqt1/detection_loss‚1.7544951841496585 /loss/FRt2/detection_loss‚1.7745346360060634 .loss/FBt1/detection_loss‚3.336607962846756 _runtime‚3475.313982462 /loss/FRt1/detection_loss‚2.3010397223489627 2loss/SRqqt3/assignment_loss‚1.5134613276631743 0loss/FBt4/assignment_loss‚0.3607361644506447 % _timestamp‚1.7884513837596393e+09 0loss/FBt2/assignment_loss‚0.2191777578385724 /loss/FRt3/detection_loss‚2.1120918851632338 1loss/SRqqt3/detection_loss‚1.8738733193388692 %loss/total_loss‚45.52641493310001 0loss/FRt1/assignment_loss‚2.9606961011886597 0loss/FRt2/assignment_loss‚1.7396531190190996 0loss/FRt3/assignment_loss‚3.2472964708621683 0loss/FBt1/assignment_loss‚0.1153285635324816~Hß`•Ú /loss/FBt2/detection_loss‚0.6658392310673664 /loss/FBt3/detection_loss‚2.9753264018467496 trainer/global_step‚4499 1loss/SRqqt2/assignment_loss‚1.006911070406204 .loss/FRt4/detection_loss‚3.514742116133372 1loss/SRqqt2/detection_loss‚1.2502818417269737 epoch‚3 0loss/FRt4/assignment_loss‚3.6294554670651755 1loss/SRqqt1/assignment_loss‚1.496059173293281 1loss/SRqqt1/detection_loss‚1.7544951841496585 /loss/FRt2/detection_loss‚1.7745346360060634 .loss/FBt1/detection_loss‚3.336607962846756 _runtime‚3475.313982462 /loss/FRt1/detection_loss‚2.3010397223489627 2loss/SRqqt3/assignment_loss‚1.5134613276631743 0loss/FBt4/assignment_loss‚0.3607361644506447 % _timestamp‚1.7884513837596393e+09 0loss/FBt2/assignment_loss‚0.2191777578385724 /loss/FRt3/detection_loss‚2.1120918851632338 1loss/SRqqt3/detection_loss‚1.8738733193388692 %loss/total_loss‚45.52641493310001 0loss/FRt1/assignment_loss‚2.9606961011886597 0loss/FRt2/assignment_loss‚1.7396531190190996 0loss/FRt3/assignment_loss‚3.2472964708621683 0loss/FBt1/assignment_loss‚0.1153285635324816 /loss/FBt4/detection_loss‚2.8994919300079345 2loss/SRqqt4/assignment_loss‚1.9603838950395585 1loss/FBt3/assignment_loss‚0.36427606602332424 _step‚368 0loss/SRqqt4/detection_loss‚2.454655526081721ðá:ñ–‚(:… ¼´æÔý²óÕ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ-Ü <—‚(:± ¼´æÔäÆØ disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.94140625% disk.nvme0n1.out‚14116.9638671875 disk.drbd1777.out‚0" disk.dm-0.out‚14116.9638671875 proc.cpu.threads‚17 network.sent‚ 9.881395e+06' proc.memory.availableMB‚ 556.8046875 disk.dm-1.out‚ 131.71875 disk.rbd1.out‚ 1.83984375 cpu‚25.324519066598086 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 131.71875% disk./.usageGB‚1110.2265014648438 disk.dm-1.in‚ 0.01171875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.94140625* proc.memory.percent‚17.064857482910156 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.out‚ 79.4873046875 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2795.90625 disk.sda.out‚ 131.71875' disk.nvme0c0n1.out‚14116.9638671875 network.recv‚ 1.259857e+06) disk./.usagePercent‚74.50823205538741 disk.sda2.in‚0 disk.dm-0.in‚ 162.94140625$ memory_percent‚96.60153388977051Qé:?‘˜‚(:† Ë´æÔÊÅÕ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.142 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¶’ž>™‚(:³ Ë´æÔçý”Ù network.sent‚ 9.98935e+06 disk.rbd1.out‚ 1.83984375 disk.rbd0.out‚0# disk.nvme0n1.out‚14117.50390625 proc.memory.rssMB‚ 2795.90625 disk.sda.out‚ 132.33203125 disk.sda3.in‚ 0.01171875 disk.dm-1.out‚ 132.33203125$ memory_percent‚96.60019874572754 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 79.4873046875 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875% disk./.usageGB‚1110.2265586853027" disk.nvme0c0n1.in‚ 162.94140625% disk.nvme0c0n1.out‚14117.50390625 disk.drbd1777.in‚0 disk.dm-0.out‚14117.50390625' proc.memory.availableMB‚ 557.0234375 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 132.33203125 disk.rbd0.in‚0 disk.dm-0.in‚ 162.94140625* proc.memory.percent‚17.064857482910156 network.recv‚ 1.265833e+06 disk.rbd3.out‚0 disk.sda2.in‚0 cpu‚25.323198505926843) disk./.usagePercent‚74.50823589550043 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 disk.sda2.out‚0 proc.cpu.threads‚17Bêà’š‚(:‡ Ú´æÔиÇÕ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.871 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àã¸C›‚(:¸ Ú´æÔ„ñŽØ disk.rbd3.out‚0 disk.sda3.out‚ 132.671875# disk.dm-0.out‚14220.16357421875 disk.dm-1.out‚ 132.671875) disk./.usagePercent‚74.50855078476755 disk.nvme0n1.in‚ 162.94140625& disk.nvme0n1.out‚14220.16357421875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 132.671875 disk.rbd0.out‚0 cpu‚25.322054901144263 network.recv‚ 1.270465e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda1.out‚0$ memory_percent‚96.60165309906006( proc.memory.availableMB‚ 556.78515625 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 network.sent‚ 9.995783e+06" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625 proc.memory.rssMB‚ 2795.90625* proc.memory.percent‚17.064857482910156% disk./.usageGB‚1110.2312507629395( disk.nvme0c0n1.out‚14220.16357421875 disk.rbd2.out‚ 79.6279296875 proc.cpu.threads‚17 disk.rbd1.in‚0Õ¡v2œ- _wandbruntime‚3511  _runtime‚3511L–ÇÎÇÁ & lr-AdamW/pg1‚0.0001446599281903571 & lr-AdamW/pg2‚0.0001446599281903571 trainer/global_step‚4549 % _timestamp‚1.7884514196568427e+09 _runtime‚3511.211409612 _step‚369ñõcôÌžÆ & lr-AdamW/pg2‚0.0001446599281903571 trainer/global_step‚4549 % _timestamp‚1.7884514196568427e+09 _runtime‚3511.211409612 _step‚369 & lr-AdamW/pg1‚0.0001446599281903571ñq¢8<2Ÿ- _wandbruntime‚3511  _runtime‚3511ÈH˜ÃÝ' ×O BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267183288374272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151493805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 _runtime‚3511.212214428 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3799173686784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60049810 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21650898 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3785191732478 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884514196576536e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256499997979648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65822732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256500015019008 _step‚370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291675112446976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267183288374272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85651035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85671073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267182614008832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3985087132160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60050256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263384114687488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85671073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3799173686784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3798517324288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952¤wæÜâ'¡ÜO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263384114687488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85671073 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3799173686784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3798517324288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267183288374272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151493805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 _runtime‚3511.212214428 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3799173686784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60049810 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21650898 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3785191732478 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884514196576536e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256499997979648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65822732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256500015019008 _step‚370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38399356 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291675112446976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267183288374272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85651035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260284535274370 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85671073 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267182614008832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287690001374720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3985087132160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60050256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844òE(e2¢- _wandbruntime‚3512  _runtime‚3512\~QbØ £àM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267239786475520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 _step‚371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506235 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytesÙRÙM.large_pool.freed‚263440449723392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884514206004553e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3799336752128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3985944218624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 3512.15547741 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3799993127936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287751652112384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291737424331776 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ɤë&¤åM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267240460854272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3985944218624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 3512.15547741 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65836746 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291737620271104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3799993127936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287751652112384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3985772219392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85669489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291737424331776 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267239786475520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 _step‚371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506235 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884514206004553e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263440449723392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3799336752128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872ó¾é2¥- _wandbruntime‚3512  _runtime‚3512o=$6W¦Ñ 0loss/FRt3/assignment_loss‚3.2339824124918146 2loss/SRqqt2/assignment_loss‚0.9265121773310137 2loss/SRqqt3/assignment_loss‚1.9959974511275216 -loss/FRt4/detection_loss‚3.47782964455454 epoch‚3 /loss/FRt4/assignment_loss‚4.363635797249644 /loss/FRt2/detection_loss‚1.6452563133884648 1loss/SRqqt1/detection_loss‚1.7295668615171542 0loss/SRqqt3/detection_loss‚1.722866024053286 0loss/SRqqt4/detection_loss‚2.471201972828971 0loss/FBt2/assignment_loss‚0.2695265864884764 %loss/total_loss‚44.77301247569277 trainer/global_step‚4549 0loss/FBt3/assignment_loss‚0.3364840266960008 /loss/FRt2/assignment_loss‚1.561308680987749 .loss/FRt1/detection_loss‚2.317142925997998 1loss/SRqqt2/detection_loss‚1.2668165487417005 /loss/FBt2/detection_loss‚1.1610882108919878 _step‚372 2loss/SRqqt4/assignment_loss‚2.0880881448586783 0loss/FBt4/assignment_loss‚0.3496190424148853 /loss/FRt3/detection_loss‚2.0610108880673423 /loss/FBt3/detection_loss‚2.2842568159103394 # _timestamp‚1.78845142060232e+09 _runtime‚3512.156600926 /loss/FBt1/detection_loss‚2.6370447754859923 /loss/FBt4/detection_loss‚2.8450282445320716 0loss/FRt1/assignment_loss‚2.4388260232641343 2loss/SRqqt1/assignment_loss‚1.3340467819764923 1loss/FBt1/assignment_loss‚0.25587612483650446Cˆmd\§Ö 0loss/FBt3/assignment_loss‚0.3364840266960008 /loss/FRt2/assignment_loss‚1.561308680987749 .loss/FRt1/detection_loss‚2.317142925997998 1loss/SRqqt2/detection_loss‚1.2668165487417005 /loss/FBt2/detection_loss‚1.1610882108919878 _step‚372 2loss/SRqqt4/assignment_loss‚2.0880881448586783 0loss/FBt4/assignment_loss‚0.3496190424148853 /loss/FRt3/detection_loss‚2.0610108880673423 /loss/FBt3/detection_loss‚2.2842568159103394 # _timestamp‚1.78845142060232e+09 _runtime‚3512.156600926 /loss/FBt1/detection_loss‚2.6370447754859923 /loss/FBt4/detection_loss‚2.8450282445320716 0loss/FRt1/assignment_loss‚2.4388260232641343 2loss/SRqqt1/assignment_loss‚1.3340467819764923 1loss/FBt1/assignment_loss‚0.25587612483650446 0loss/FRt3/assignment_loss‚3.2339824124918146 2loss/SRqqt2/assignment_loss‚0.9265121773310137 2loss/SRqqt3/assignment_loss‚1.9959974511275216 -loss/FRt4/detection_loss‚3.47782964455454 epoch‚3 /loss/FRt4/assignment_loss‚4.363635797249644 /loss/FRt2/detection_loss‚1.6452563133884648 1loss/SRqqt1/detection_loss‚1.7295668615171542 0loss/SRqqt3/detection_loss‚1.722866024053286 0loss/SRqqt4/detection_loss‚2.471201972828971 0loss/FBt2/assignment_loss‚0.2695265864884764 %loss/total_loss‚44.77301247569277 trainer/global_step‚4549ô[Úq…“¨‚(:ˆ é´æÔúÙÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.299' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏýG©‚(:¼ é´æÔ²þØ disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 81.1396484375" disk.nvme0c0n1.in‚ 162.94140625 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14221.06884765625 network.recv‚ 1.27439e+06) disk./.usagePercent‚74.50855360085042 disk.dm-0.in‚ 162.94140625 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625% disk./.usageGB‚1110.2312927246094* proc.memory.percent‚17.064857482910156# disk.dm-0.out‚14221.06884765625 disk.dm-1.in‚ 0.01171875( proc.memory.availableMB‚ 556.76953125 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 133.28515625& disk.nvme0n1.out‚14221.06884765625# memory_percent‚96.6017484664917 disk.sda3.out‚ 133.28515625 disk.rbd0.in‚0 disk.rbd2.in‚0 network.sent‚ 1.0001299e+07 disk.dm-1.out‚ 133.28515625 disk.sda1.in‚0 proc.memory.rssMB‚ 2795.90625 disk.drbd1777.out‚0 disk.sda2.in‚0 cpu‚25.32089282904234Wjè$“ª‚(:ˆ ø´æÔÓçþÕ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚164.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çl]BD«‚(:¹ ø´æÔ¤„‚Ù disk.sda.out‚ 133.76953125 disk.sda1.out‚0& disk.nvme0n1.out‚14318.52978515625 cpu‚25.319822686584462 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 133.76953125 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1110.2312965393066 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 162.94140625 proc.cpu.threads‚17 network.sent‚ 1.0110269e+07 disk.sda3.in‚ 0.01171875) disk./.usagePercent‚74.50855385685796 disk.dm-1.out‚ 133.76953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625$ memory_percent‚96.60186767578125" proc.memory.availableMB‚556.75* proc.memory.percent‚17.064857482910156 disk.rbd3.in‚0 network.recv‚ 1.280893e+06 disk.dm-0.in‚ 162.94140625 disk.sda.in‚ 0.01171875 disk.rbd2.out‚ 81.2177734375( disk.nvme0c0n1.out‚14318.52978515625 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2795.90625# disk.dm-0.out‚14318.52978515625ó‘$u2¬- _wandbruntime‚3548  _runtime‚3548¹š(ÿÉ­Ã _runtime‚3548.076109219 _step‚373 ' lr-AdamW/pg1‚0.00014356152752260123 ' lr-AdamW/pg2‚0.00014356152752260123 trainer/global_step‚4599 % _timestamp‚1.7884514565217083e+09Œ€BÎ®È ' lr-AdamW/pg1‚0.00014356152752260123 ' lr-AdamW/pg2‚0.00014356152752260123 trainer/global_step‚4599 % _timestamp‚1.7884514565217083e+09 _runtime‚3548.076109219 _step‚373õ‡æ½?2¯- _wandbruntime‚3548  _runtime‚3548:Ëò'°×O RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266201748626944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86577087 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38807756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290772538257920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚4599 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 _runtime‚3548.077023342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270041251757056 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884514565224533e+09 UGDeviceStatsMonitor.on_train_batch_start/reqZÃVÉuested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0cÖ©â'±ÜO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66523532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38807756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290772538257920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚4599 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 _runtime‚3548.077023342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270041251757056 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884514565224533e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266201748626944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3840159492608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86577087 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884514572184489e+09 _runtime‚3548.773444831 _step‚375 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263126297707302 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21881019 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66537546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3840322557952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263126969184626 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86595541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294863114573312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266258119668736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294862927022592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0î£ ì&´æM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38815926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38815924 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21881019 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66537546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3840322557952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263126969184626 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86595541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294863114573312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266258119668736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290834188995584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294862927022592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826843507058 trainer/global_step‚4599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884514572184489e+09 _runtime‚3548.773444831 _step‚375 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263126297707302 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0÷ƒ$Ã2µ- _wandbruntime‚3548  _runtime‚3548„Z\¶Ö 0loss/FBt4/assignment_loss‚0.4969489984206281 0loss/FRt3/assignment_loss‚2.9814665866833105 2loss/SRqqt3/assignment_loss‚1.5878324564546347 /loss/FRt4/detection_loss‚2.7498236195794465 /loss/FBt3/detection_loss‚2.2967881697874803 % _timestamp‚1.7884514572201822e+09 /loss/FRt1/assignment_loss‚3.437062844634056 /loss/FRt1/detection_loss‚2.5257290162146093 2loss/SRqqt1/assignment_loss‚1.7506437979086682 /loss/FRt3/detection_loss‚2.2519525762866524 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚4.367378650040462 2loss/SRqqt2/assignment_loss‚1.1975231601575773 0loss/FBt1/assignment_loss‚0.5457087529874924 1loss/FBt2/assignment_loss‚0.18698243890995614 0loss/FBt3/assignment_loss‚0.6824778053336419 1loss/SRqqt4/detection_loss‚2.3226253747940064 /loss/FBt2/detection_loss‚0.6363464541080603 1loss/SRqqt3/detection_loss‚1.7771326154470444 /loss/FBt1/detection_loss‚2.9995319181018405 .loss/FBt4/detection_loss‚2.610939730616177 1loss/SRqqt4/assignment_loss‚2.254543846845627 /loss/FRt2/detection_loss‚1.6530285974104817 1loss/SRqqt1/detection_loss‚1.9251574465522059 _runtime‚3548.774449523 0loss/FRt2/assignment_loss‚1.6583552945118685 1loss/SRqqt2/detection_loss‚0.8883812871599008 %loss/total_loss‚45.78436143894582 epoch‚3 _step‚376”HWFŠ·Û 2loss/SRqqt1/assignment_loss‚1.7506437979086682 /loss/FRt3/detection_loss‚2.2519525762866524 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚4.367378650040462 2loss/SRqqt2/assignment_loss‚1.1975231601575773 0loss/FBt1/assignment_loss‚0.5457087529874924 1loss/FBt2/assignment_loss‚0.18698243890995614 0loss/FBt3/assignment_loss‚0.6824778053336419 1loss/SRqqt4/detection_loss‚2.3226253747940064 /loss/FBt2/detection_loss‚0.6363464541080603 1loss/SRqqt3/detection_loss‚1.7771326154470444 /loss/FBt1/detection_loss‚2.9995319181018405 .loss/FBt4/detection_loss‚2.610939730616177 1loss/SRqqt4/assignment_loss‚2.254543846845627 /loss/FRt2/detection_loss‚1.6530285974104817 1loss/SRqqt1/detection_loss‚1.9251574465522059 _runtime‚3548.774449523 0loss/FRt2/assignment_loss‚1.6583552945118685 1loss/SRqqt2/detection_loss‚0.8883ÊjÍÚ×812871599008 %loss/total_loss‚45.78436143894582 epoch‚3 _step‚376 0loss/FBt4/assignment_loss‚0.4969489984206281 0loss/FRt3/assignment_loss‚2.9814665866833105 2loss/SRqqt3/assignment_loss‚1.5878324564546347 /loss/FRt4/detection_loss‚2.7498236195794465 /loss/FBt3/detection_loss‚2.2967881697874803 % _timestamp‚1.7884514572201822e+09 /loss/FRt1/assignment_loss‚3.437062844634056 /loss/FRt1/detection_loss‚2.5257290162146093øm x’¸‚(:‡ ‡µæÔ͘…Ö gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚100.785' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.157 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®êÒC¹‚(:¸ ‡µæÔÒ—õÙ disk.sda3.in‚ 0.01171875% disk.nvme0n1.out‚14319.1142578125 network.sent‚ 1.0115461e+07 network.recv‚ 1.28432e+06* proc.memory.percent‚17.064857482910156" disk.nvme0c0n1.in‚ 162.94140625' disk.nvme0c0n1.out‚14319.1142578125 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 134.24609375 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚14319.1142578125 proc.memory.rssMB‚ 2795.90625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625' proc.memory.availableMB‚ 556.6015625 disk.drbd1777.in‚0) disk./.usagePercent‚74.49192667952411 disk.dm-1.out‚ 134.24609375 disk.rbd3.in‚0 disk.sda.out‚ 134.24609375 disk.rbd0.in‚0 disk.rbd2.out‚ 81.2177734375$ memory_percent‚96.60277366638184% disk./.usageGB‚1109.9835395812988 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 cpu‚25.3186792454909 disk.sda2.in‚0 proc.cpu.threads‚17'17°žº‚(:“ –µæÔ®µ×Õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.947' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.589399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>÷’cH»‚(:½ –µæÔãƨØ" disk.dm-0.out‚14580.1220703125 network.sent‚ 1.0224784e+07 network.recv‚ 1.291289e+06) disk./.usagePercent‚74.43899328170693% disk./.usageGB‚1109.1947937011719 disk.rbd1.in‚0 disk.rbd2.out‚81.32763671875% disk.nvme0n1.out‚14580.1220703125 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625 cpu‚25.317581728138837 disk.sda.out‚ 134.91796875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚14580.1220703125$ memory_percent‚96.60351276397705 proc.cpu.threads‚17 disk.sda3.out‚ 134.91796875 disk.rbd2.in‚0* proc.memory.percent‚17.064857482910156 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.out‚0( proc.memory.availableMB‚ 556.48046875 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.dm-1.out‚ 134.91796875 disk.sda3.in‚ 0.01171875G;]&ž¼‚(:“ ¥µæÔœûÿÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.809' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.561799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¯Æ(I½‚(:¾ ¥µæÔíÁþ× disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚14580.72900390625 disk.drbd1777.in‚0# disk.dm-0.out‚14580.72900390625 disk.sda2.out‚0 cpu‚25.31647489138158 proc.cpu.threads‚17 disk.sda.out‚ 135.47265625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 162.94140625' proc.memory.availableMB‚ 556.3671875 proc.memory.rssMB‚ 2795.90625 disk.sda3.out‚ 135.47265625 disk.dm-1.out‚ 135.47265625$ memory_percent‚96.60420417785645 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚14580.72900390625 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚81.32763671875" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 network.recv‚ 1.302444e+06) disk./.usagePercent‚74.43897049703637* proc.memory.percent‚17.064857482910156 network.sent‚ 1.0233968e+07% disk./.usageGB‚1109.19445419311528-{2¾- _wandbruntime‚3585  _runtime‚3585þ…É¿Ã ' lr-AdamW/pg2‚0.00014245241114115286 trainer/global_step‚4649 % _timestamp‚1.7884514938594666e+09 _runtime‚3585.413875118 _step‚377 ' lr-AdamW/pg1‚0.00014245241114115286>×ÎÀÈ ' lr-AdamW/pg2‚0.00014245241114115286 trainer/global_step‚4649 % _timestamp‚1.7884514938594666e+09 _runtime‚3585.413875118 _step‚377 ' lr-AdamW/pg1‚0.00014245241114115286ù,˜T½2Á- _wandbruntime‚3585  _runtime‚3585fGú…Ü'ÂÖO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 _runtime‚3585.414818606 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4070701693440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788451493860216e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87523177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3881145298432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 trainer/global_step‚4649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154727469 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261990411035648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67224330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330gŸ×á'ÃÛO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272900563870720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272900563870720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272899889505280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269019418572288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22102878 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293855075141120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4070863220224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3880488935936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297925776834560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261990411035648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3866862411294 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67224330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67224330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 _runtime‚3585.414818606 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4070701693440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788451493860216e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87523177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3881145298432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67224332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22103296 trainer/global_step‚4649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154727469 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4071566511104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265913011052034 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚4649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 # _timestamp‚1.78845149454672e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 3586.10187575 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytesŸL¢¦° .small_pool.freed‚ 3881308363776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3881308363776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚379 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414Dˆé&ÆãM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67238348 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61332168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262045315166208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272957736350720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4071566511104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265913011052034 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚4649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 # _timestamp‚1.78845149454672e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 3586.10187575 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3867678846466 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3881308363776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881964739584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269075771611136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚379 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928û´jÆ2Ç- _wandbruntime‚3586  _runtime‚35866Ø¡Ã[ÈÕ 1loss/SRqqt1/assignment_loss‚2.103496494618329 2loss/SRqqt2/assignment_loss‚1.0880378181843653 /loss/FRt3/detection_loss‚2.3856790221255757 1loss/SRqqt2/detection_loss‚0.9618787703954655 %loss/total_loss‚45.22875804444709 /loss/FRt1/detection_loss‚2.4222599702576795 -loss/FBt4/detection_loss‚2.67309707403183 trainer/global_step‚4649 _step‚380 0loss/FRt1/assignment_loss‚3.1365183194478354 /loss/FRt4/assignment_loss‚4.021007609367371 2loss/SRqqt4/assignment_loss‚1.8648512094761387 1loss/FBt2/assignment_loss‚0.27647712229703236 /loss/FRt4/detection_loss‚3.1112878036499025 1loss/SRqqt3/detection_loss‚1.8506113821268082 0loss/FRt3/assignment_loss‚3.3495607155820597 /loss/FRt2/detection_loss‚1.6256941477768123 1loss/SRqqt1/detection_loss‚1.9875084646046162 1loss/SRqqt4/detection_loss‚2.4195203347639604 /loss/FBt2/detection_loss‚0.7948085189599078 _runtime‚3586.103435649 0loss/FBt4/assignment_loss‚0.4333431167261936 0loss/FRt2/assignment_loss‚1.2598311347620827 .loss/FBt3/detection_loss‚2.726462334394455 epoch‚3 2loss/SRqqt3/assignment_loss‚1.2740671406686306 0loss/FBt3/assignment_loss‚0.8145045228302479 /loss/FBt1/detection_loss‚1.9633671939373016 0loss/FBt1/assignment_loss‚0.6848878234624862 % _timestamp‚1.7884514945490978e+09^.ò`ÉÚ 0loss/FRt2/assignment_loss‚1.2598311347620827 .loss/FBt3/detection_loss‚2.726462334394455 epoch‚3 2loss/SRqqt3/assignment_loss‚1.2740671406686306 0loss/FBt3/assignment_loss‚0.8145045228302479 /loss/FBt1/detection_loss‚1.9633671939373016 0loss/FBt1/assignment_loss‚0.6848878234624862 % _timestamp‚1.7884514945490978e+09 1loss/SRqqt1/assignment_loss‚2.103496494618329 2loss/SRqqt2/assignment_loss‚1.0880378181843653 /loss/FRt3/detection_loss‚2.3856790221255757 1loss/SRqqt2/detection_loss‚0.9618787703954655 %loss/total_loss‚45.22875804444709 /loss/FRt1/detection_loss‚2.4222599702576795 -loss/FBt4/detection_loss‚2.67309707403183 trainer/global_step‚4649 _step‚380 0loss/FRt1/assignment_loss‚3.1365183194478354 /loss/FRt4/assignment_loss‚4.021007609367371 2loss/SRqqt4/assignment_loss‚1.8648512094761387 1loss/FBt2/assignment_loss‚0.27647712229703236 /loss/FRt4/detection_loss‚3.1112878036499025 1loss/SRqqt3/detection_loss‚1.8506113821268082 0loss/FRt3/assignment_loss‚3.3495607155820597 /loss/FRt2/detection_loss‚1.6256941477768123 1loss/SRqqt1/detection_loss‚1.9875084646046162 1loss/SRqqt4/detection_loss‚2.4195203347639604 /loss/FBt2/detection_loss‚0.7948085189599078 _runtime‚3586.103435649 0loss/FBt4/assignment_loss‚0.4333431167261936ü=Ap‡“Ê‚(:ˆ ´µæÔëÀðÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãà*wEË‚(:º ´µæÔí¹Ø disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd2.out‚82.91748046875& disk.nvme0n1.out‚14581.03759765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 proc.memory.rssMB‚ 2795.90625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.out‚ 136.16015625 network.recv‚ 1.30688e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14581.03759765625 disk.rbd1.in‚0 cpu‚25.315439659154745 disk.sda3.out‚ 136.16015625 disk.rbd2.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 136.16015625 disk.sda1.out‚0# disk.dm-0.out‚14581.03759765625 disk.sda2.out‚0$ memory_percent‚96.60415649414062* proc.memory.percent‚17.064857482910156 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.in‚0# proc.memory.availableMB‚556.375 network.sent‚ 1.0240219e+07) disk./.usagePercent‚74.43900864215898% disk./.usageGB‚1109.1950225830078 disk.sda3.in‚ 0.01171875ÙêcZ“Ì‚(:ˆ õæÔÀœÐÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.9418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—aƹ@Í‚(:µ õæÔõËê× network.sent‚ 1.0348411e+07 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.out‚14581.73828125 disk.rbd3.in‚0 disk.sda.out‚ 136.59765625 cpu‚25.31434998253316 disk.sda1.out‚0 disk.rbd2.out‚82.91748046875 disk.sda2.out‚0 disk.rbd1.out‚ 1.83984375 network.recv‚ 1.312919e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 136.59765625' proc.memory.availableMB‚ 556.5546875 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda3.out‚ 136.59765625 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625) disk./.usagePercent‚74.43900992219665# disk.nvme0n1.out‚14581.73828125 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17% disk./.usageGB‚1109.1950416564941% disk.nvme0c0n1.out‚14581.73828125 disk.sda2.in‚0$ memory_percent‚96.60305976867676* proc.memory.percent‚17.064857482910156è¾O2Î- _wandbruntime‚3621  _runtime‚3621Ug|¨ÇÏÁ % _timestamp‚1.7884515303685763e+09 _runtime‚3621.923107055 _step‚381 & lr-AdamW/pg1‚0.0001413328101842035 & lr-AdamW/pg2‚0.0001413328101842035 trainer/global_step‚4699¢¾ØÌÐÆ & lr-AdamW/pg1‚0.0001413328101842035 & lr-AdamW/pg2‚0.0001413328101842035 trainer/global_step‚4699 % _timestamp‚1.7884515303685763e+09 _runtime‚3621.923107055 _step‚381ý‰˜Œl2Ñ- _wandbruntime‚3621  _runtime‚3621†ÌõŒÝ'Ò×O ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271837070514688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296937635964416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3907043312946 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67925130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3921474741760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚\)VP67925130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3921474741760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4113864276992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264735617563648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22329462 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760þe‡…2Ô- _wandbruntime‚3622  _runtime‚3622Á[D,ç&ÕáM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275815699720192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚383 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67939146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3907859747910 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39632726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 _runtime‚3622.620617771 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67939148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67939148 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3922294169600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0V&û¾ì&ÖæM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88467709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67939148 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3922294169600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275815699720192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301113828401664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚383 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67939146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3907859747910 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39632726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 _runtime‚3622.620617771 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67939148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791ÿjÏ2×- _wandbruntime‚3622  _runtime‚3622>æØZØÔ 1loss/FBt3/assignment_loss‚0.43980795294046404 /loss/FRt2/detection_loss‚1.4651962706675896 /loss/FRt3/detection_loss‚2.3380781001514856 % _timestamp‚1.7884515310674853e+09 0loss/SRqqt1/detection_loss‚1.884534592520107 1loss/SRqqt3/detection_loss‚1.8112169295026546 .loss/FBt4/detection_loss‚3.467053711414337 epoch‚3 2loss/SRqqt2/assignment_loss‚1.1757050783994296 2loss/SRqqt4/assignment_loss‚1.9942301789234425 0loss/FBt1/assignment_loss‚0.5090257674455643 0loss/FBt4/assignment_loss‚0.6288482633729776 .loss/FRt4/detection_loss‚3.041584259477155 0loss/SRqqt4/detection_loss‚2.362182634657827 /loss/FRt3/assignment_loss‚3.072421918974982 2loss/SRqqt1/assignment_loss‚1.6288443591107022 2loss/SRqqt3/assignment_loss‚1.2383865838249524 /loss/FBt1/detection_loss‚3.7558748026688895 _step‚384 0loss/FRt2/assignment_loss‚1.6156505685586195 1loss/SRqqt2/detection_loss‚1.2231958635151385 .loss/FBt3/detection_loss‚2.665438675880432 trainer/global_step‚4699 0loss/FRt1/assignment_loss‚3.5515238401052116 0loss/FRt4/assignment_loss‚3.8632930434983352 /loss/FBt2/detection_loss‚0.5475209063598984 /loss/FRt1/detection_loss‚2.7367479732713185 _runtime‚3622.621837549 0loss/FBt2/assignment_loss‚0.1907491109363147 %loss/total_loss‚47.20711138617783/¨Œo_ÙÙ 0loss/FBt2/assignment_loss‚0.1907491109363147 %loss/total_loss‚47.20711138617783 1loss/FBt3/assignment_loss‚0.43980795294046404 /loss/FRt2/detection_loss‚1.4651962706675896 /loss/FRt3/detection_loss‚2.3380781001514856 % _timestamp‚1.7884515310674853e+09 0loss/SRqqt1/detection_loss‚1.884534592520107 1loss/SRqqt3/detection_loss‚1.8112169295026546 .loss/FBt4/detection_loss‚3.467053711414337 epoch‚3 2loss/SRqqt2/assignment_loss‚1.1757050783994296 2loss/SRqqt4/assignment_loss‚1.9942301789234425 0loss/FBt1/assignment_loss‚0.5090257674455643 0loss/FBt4/assignment_loss‚0.6288482633729776 .loss/FRt4/detection_loss‚3.041584259477155 0loss/SRqqt4/detection_loss‚2.362182634657827 /loss/FRt3/assignment_loss‚3.072421918974982 2loss/SRqqt1/assignment_loss‚1.6288443591107022 2loss/SRqqt3/assignment_loss‚1.2383865838249524 /loss/FBt1/detection_loss‚3.7558748026688895 _step‚384 0loss/FRt2/assignment_loss‚1.6156505685586195 1loss/SRqqt2/detection_loss‚1.2231958635151385 .loss/FBt3/detection_loss‚2.665438675880432 trainer/global_step‚4699 0loss/FRt1/assignment_loss‚3.5515238401052116 0loss/FRt4/assignment_loss‚3.8632930434983352 /loss/FBt2/detection_loss‚0.5475209063598984 /loss/FRt1/detection_loss‚2.7367479732713185 _runtime‚3622.621837549€¾Nµ¥žÚ‚(:“ ÒµæÔžÚÌÕ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.374799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â-6¢AÛ‚(:¶ ÒµæÔ§®ŒÙ& disk.nvme0c0n1.out‚14582.162109375 network.recv‚ 1.324253e+06 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0* proc.memory.percent‚17.064857482910156" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda.in‚ 0.01171875$ disk.nvme0n1.out‚14582.162109375! disk.dm-0.out‚14582.162109375 disk.sda2.out‚0 disk.rbd1.out‚ 1.83984375$ memory_percent‚96.60308361053467% disk./.usageGB‚1109.1950569152832 disk.sda3.in‚ 0.01171875 proc.memory.rssMB‚ 2795.90625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 137.06640625 proc.cpu.threads‚17 network.sent‚ 1.0358197e+07) disk./.usagePercent‚74.43901094622677 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 83.0859375 disk.drbd1777.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 556.55078125 disk.sda3.out‚ 137.06640625 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.out‚ 137.06640625 cpu‚25.313179039418753#1tB’Ü‚(:‡ áµæÔ¯çÕ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚94.572' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÇ* Ý‚(:¹ áµæÔ»·úØ disk.sda3.out‚ 137.61328125& disk.nvme0n1.out‚14582.81591796875 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 137.61328125 disk.sda.out‚ 137.61328125& proc.memory.availableMB‚ 556.421875! proc.memory.rssMB‚ 2795.984375 disk.rbd1.out‚ 1.83984375 disk.rbd2.in‚0( disk.nvme0c0n1.out‚14582.81591796875 disk.dm-0.in‚ 162.94140625 disk.sda.in‚ 0.01171875# memory_percent‚96.6038703918457 network.recv‚ 1.330543e+06 disk.nvme0n1.in‚ 162.94140625# disk.dm-0.out‚14582.81591796875 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.43901222626445 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875) proc.memory.perceÖ†¬ØBnt‚17.06533432006836 cpu‚25.312105280539924" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 83.0859375 network.sent‚ 1.0466484e+07% disk./.usageGB‚1109.1950759887695 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0£qÇp2Þ- _wandbruntime‚3658  _runtime‚3658ñ\åµÉßà ' lr-AdamW/pg1‚0.00014020295797491448 ' lr-AdamW/pg2‚0.00014020295797491448 trainer/global_step‚4749 % _timestamp‚1.7884515670350657e+09 _runtime‚3658.589436608 _step‚385k¤†âÎàÈ ' lr-AdamW/pg1‚0.00014020295797491448 ' lr-AdamW/pg2‚0.00014020295797491448 trainer/global_step‚4749 % _timestamp‚1.7884515670350657e+09 _runtime‚3658.589436608 _step‚385€¿û¹2á- _wandbruntime‚3658  _runtime‚3658”@‰ÑÝ'â×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68625932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274654722457088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3948533090110 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278617165001728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚3658.590597465 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278617839367168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40032956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40032958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300020148907520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300020172847616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156807825920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3962460547584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267480841131008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚386 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3947878652354 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274654704454144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62589644 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68625932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963116910080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3948533090110 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278617165001728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚3658.590597465 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278617839367168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40032956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40032958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300020148907520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300020172847616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156807825920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3962460547584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267480841131008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚386 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3947878652354 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274654704454144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62589644 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68625932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3963116910080 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68639946 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963279975424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274711075495936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62602281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚387 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40041124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884515677140017e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4749 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271485094786850 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157490690048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3659.268986902 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282àÅ¥ææM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267535728222208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884515677140017e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4749 ^JDeviceStatsMoni¥Ål¶G!tor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68639946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271485094786850 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271484423309526 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157490690048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3659.268986902 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68639946 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963279975424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274711075495936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62602281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚387 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40041124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E2ç- _wandbruntime‚3659  _runtime‚3659×ì£ÙYèÓ .loss/FRt4/detection_loss‚2.947796391589301 1loss/SRqqt4/detection_loss‚2.6266828974088035 /loss/FBt3/detection_loss‚1.9046248992284138 /loss/FBt4/detection_loss‚2.7681876942515373 0loss/SRqqt3/detection_loss‚1.961274653356126 trainer/global_step‚4749 0loss/FBt2/assignment_loss‚0.2368208792915216 0loss/FBt3/assignment_loss‚0.7792692709093293 0loss/FBt4/assignment_loss‚0.7261595465242827 /loss/FRt2/detection_loss‚1.7697452205825936 /loss/FBt2/detection_loss‚0.5976674717230093 0loss/FRt4/assignment_loss‚4.5448905144418985 2loss/SRqqt2/assignment_loss‚1.0023275438362538 2loss/SRqqt4/assignment_loss‚2.2656743983427683 .loss/FRt1/detection_loss‚2.696655197814107 .loss/FRt3/detection_loss‚2.538808384019396 1loss/SRqqt1/detection_loss‚2.3926708159201286 _runtime‚3659.270247597 /loss/FBt1/detection_loss‚2.7900442123413085 epoch‚3 % _timestamp‚1.7884515677158961e+09 _step‚388 /loss/FRt1/assignment_loss‚3.389779457449913 0loss/FRt2/assignment_loss‚1.4958527012304825 0loss/FBt1/assignment_loss‚0.5077762383967638 1loss/SRqqt2/detection_loss‚1.0562256468988176 $loss/total_loss‚47.6290364884583 0loss/FRt3/assignment_loss‚3.3207429751105932 2loss/SRqqt1/assignment_loss‚1.7736982121406233 2loss/SRqqt3/assignment_loss‚1.5356612656503281B w^éØ 0loss/FBt2/assignment_loss‚0.2368208792915216 0loss/FBt3/assignment_loss‚0.7792692709093293 0loss/FBt4/assignment_loss‚0.7261595465242827 /loss/FRt2/detection_loss‚1.7697452205825936 /loss/FBt2/detection_loss‚0.5976674717230093 0loss/FRt4/assignment_loss‚4.5448905144418985 2loss/SRqqt2/assignment_loss‚1.0023275438362538 2loss/SRqqt4/assignment_loss‚2.2656743983427683 .loss/FRt1/detection_loss‚2.696655197814107 .loss/FRt3/detection_loss‚2.538808384019396 1loss/SRqqt1/detection_loss‚2.3926708159201286 _runtime‚3659.270247597 /loss/FBt1/detection_loss‚2.7900442123413085 epoch‚3 % _timestamp‚1.7884515677158961e+09 _step‚388 /loss/FRt1/assignment_loss‚3.389779457449913 0loss/FRt2/assignment_loss‚1.4958527012304825 0loss/FBt1/assignment_loss‚0.5077762383967638 1loss/SRqqt2/detection_loss‚1.0562256468988176 $loss/total_loss‚47.6290364884583 0loss/FRt3/assignment_loss‚3.3207429751105932 2loss/SRqqt1/assignment_loss‚1.7736982121406233 2loss/SRqqt3/assignment_loss‚1.5356612656503281 .loss/FRt4/detection_loss‚2.947796391589301 1loss/SRqqt4/detection_loss‚2.6266828974088035 /loss/FBt3/detection_loss‚1.9046248992284138 /loss/FBt4/detection_loss‚2.7681876942515373 0loss/SRqqt3/detection_loss‚1.961274653356126 trainer/global_step‚4749„€`Ežê‚(:“ ðµæÔÿ­ÿÖ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ˆƒÜ>ë‚(:³ ðµæÔµÛØ proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875$ memory_percent‚96.60522937774658 network.sent‚ 1.0473607e+07% disk.nvme0c0n1.out‚14583.21484375 disk.rbd2.in‚0# disk.nvme0n1.out‚14583.21484375 disk.dm-0.out‚14583.21484375 disk.dm-1.out‚ 138.03515625 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 138.03515625( proc.memory.availableMB‚ 556.19921875 disk.sda1.out‚0) proc.memory.percent‚17.06533432006836 cpu‚25.311041991040735) disk./.usagePercent‚74.43900864215898% disk./.usageGB‚1109.1950225830078 disk.nvme0n1.in‚ 162.94140625 disk.rbd3.out‚0! proc.memory.rssMB‚ 2795.984375 disk.rbd2.out‚ 83.2265625 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 1.83984375 disk.sda3.out‚ 138.03515625" disk.nvme0c0n1.in‚ 162.94140625 network.recv‚ 1.335824e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚03£"Jžì‚(:“ ÿµæÔì³âÕ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚151.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.310599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥é6<í‚(:± ÿµæÔÉØ disk.drbd1777.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 138.6640625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 138.6640625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625! disk.dm-0.out‚14584.068359375$ memory_percent‚96.60601615905762) proc.memory.percent‚17.06533432006836 cpu‚25.30999257565712) disk./.usagePercent‚74.43900864215898% disk./.usageGB‚1109.1950225830078 disk.dm-0.in‚ 162.94140625 disk.dm-1.out‚ 138.6640625 network.sent‚ 1.0580708e+07 disk.rbd0.in‚0$ disk.nvme0n1.out‚14584.068359375 disk.sda2.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 556.0703125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0! proc.memory.rssMB‚ 2795.984375 disk.rbd2.out‚ 83.2265625 disk.dm-1.in‚ 0.01171875 network.recv‚ 1.340849e+06 disk.sda.in‚ 0.01171875 disk.sda1.in‚0& disk.nvme0c0n1.out‚14584.068359375r³Tœî‚(:‘ Ž¶æÔŸÞ³Õ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.083' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.816599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l•¬=9ï‚(:® Ž¶æÔ›ÀžÙ disk.rbd3.in‚0 disk.sda3.out‚ 139.1484375' proc.memory.availableMB‚ 556.0703125 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.rbd2.in‚0 disk.rbd2.out‚84.8125% disk./.usageGB‚1109.0284843444824! disk.dm-0.out‚14586.083984375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.345494e+06$ disk.nvme0n1.out‚14586.083984375 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 network.sent‚ 1.0587444e+07 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875& disk.nvme0c0n1.out‚14586.083984375$ memory_percent‚96.60601615905762) proc.memory.percent‚17.06533432006836 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0! proc.memory.rssMB‚ 2795.984375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 139.1484375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-1.out‚ 139.1484375) disk./.usagePercent‚74.42783212123665 disk.drbd1777.out‚0 disk.rbd0.out‚0 cpu‚25.30892698041109?tBý2ð- _wandbruntime‚3695  _runtime‚36951ˆÀ‚ÇñÁ % _timestamp‚1.7884516034792545e+09 _runtime‚3695.033670485 _step‚389 & lr-AdamW/pg1‚0.0001390630899727927 & lr-AdamW/pg2‚0.0001390630899727927 trainer/global_step‚4799Ä¢—©ÌòÆ trainer/global_step‚4799 % _timestamp‚1.7884516034792545e+09 _runtime‚3695.033670485 _step‚389 & lr-AdamW/pg1‚0.0001390630899727927 & lr-AdamW/pg2‚0.0001390630899727927…KÛ·2ó- _wandbruntime‚3695  _runtime‚3695] NâÜ'ôÖO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307302499436032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270226047659008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 _runtime‚3695.034803339 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚4799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281476477115392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22783943 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚390 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128véÁõÛO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 _runtime‚3695.034803339 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚4799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281476477115392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22783943 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69326730 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚390 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307302499436032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270226047659008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605†ךph2ö- _wandbruntime‚3695  _runtime‚3695 ·.ç&÷áM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4004265781248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90299749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69340746 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274271136654258 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270280934750208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281532975216640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277528709435392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90319813 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚4799 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884516041708574e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004265781248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 _step‚391 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3695.725756024 ž¤Öì&øæM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90319813 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚4799 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884516041708574e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004265781248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 _step‚391 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3695.725756024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4004265781248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90299749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69340746 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274271136654258 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚270280934750208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281533649595392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281532975216640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277528709435392‡26ýÓ2ù- _wandbruntime‚3695  _runtime‚36956‹@RZúÔ 2loss/SRqqt1/assignment_loss‚1.7227762898101526 1loss/FBt2/assignment_loss‚0.16586060143985695 1loss/SRqqt3/detection_loss‚2.3017856384195934 /loss/FRt4/assignment_loss‚4.168322554001441 0loss/FBt1/assignment_loss‚0.4298759445717311 0loss/FRt2/assignment_loss‚1.8015739445416432 /loss/FRt3/assignment_loss‚3.033196580410004 2loss/SRqqt2/assignment_loss‚1.0226577942595496 1loss/FBt4/assignment_loss‚0.45688072006617225 .loss/FBt2/detection_loss‚0.789137502222882 epoch‚3 0loss/FRt1/assignment_loss‚3.1253334856033326 0loss/SRqqt4/detection_loss‚3.098039762540297 _step‚392 1loss/SRqqt1/detection_loss‚1.5832179757017715 /loss/FBt3/detection_loss‚2.3365299105644226 /loss/FBt4/detection_loss‚2.6940566216196333 % _timestamp‚1.7884516041728034e+09 1loss/SRqqt3/assignment_loss‚1.547034938887852 1loss/SRqqt2/detection_loss‚1.0971888512617252 _runtime‚3695.727207589 1loss/SRqqt4/assignment_loss‚2.089421272277832 0loss/FBt3/assignment_loss‚0.6486269536499794 /loss/FRt2/detection_loss‚1.7427998741842665 .loss/FRt4/detection_loss‚3.233352326429807 %loss/total_loss‚47.14028127441854 trainer/global_step‚4799 /loss/FRt1/detection_loss‚2.2911948734521865 /loss/FRt3/detection_loss‚2.3385768972337244 /loss/FBt1/detection_loss‚3.4228399612686853àD~Ë_ûÙ /loss/FRt3/detection_loss‚2.3385768972337244 /loss/FBt1/detection_loss‚3.4228399612686853 2loss/SRqqt1/assignment_loss‚1.7227762898101526 1loss/FBt2/assignment_loss‚0.16586060143985695 1loss/SRqqt3/detection_loss‚2.3017856384195934 /loss/FRt4/assignment_loss‚4.168322554001441 0loss/FBt1/assignment_loss‚0.4298759445717311 0loss/FRt2/assignment_loss‚1.8015739445416432 /loss/FRt3/assignment_loss‚3.033196580410004 2loss/SRqqt2/assignment_loss‚1.0226577942595496 1loss/FBt4/assignment_loss‚0.45688072006617225 .loss/FBt2/detection_loss‚0.789137502222882 epoch‚3 0loss/FRt1/assignment_loss‚3.1253334856033326 0loss/SRqqt4/detection_loss‚3.098039762540297 _step‚392 1loss/SRqqt1/detection_loss‚1.5832179757017715 /loss/FBt3/detection_loss‚2.3365299105644226 /loss/FBt4/detection_loss‚2.6940566216196333 % _timestamp‚1.7884516041728034e+09 1loss/SRqqt3/assignment_loss‚1.547034938887852 1loss/SRqqt2/detection_loss‚1.0971888512617252 _runtime‚3695.727207589 1loss/SRqqt4/assignment_loss‚2.089421272277832 0loss/FBt3/assignment_loss‚0.6486269536499794 /loss/FRt2/detection_loss‚1.7427998741842665 .loss/FRt4/detection_loss‚3.233352326429807 %loss/total_loss‚47.14028127441854 trainer/global_step‚4799 /loss/FRt1/detection_loss‚2.2911948734521865ˆ¹(žžü‚(:“ ¶æÔ¸þíÕ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚152.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.581599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÙ®NBý‚(:· ¶æÔÀ±ÙØ# disk.dm-0.out‚14586.54931640625 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14586.54931640625 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 cpu‚25.307960420851625) disk./.usagePercent‚74.42771998993665" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0n1.out‚14586.54931640625 disk.sda3.out‚ 139.76953125! proc.memory.rssMB‚ 2795.984375 disk.rbd0.out‚0 disk.rbd2.out‚84.8125 disk.sda.in‚ 0.01171875 disk.sda1.in‚0) proc.memory.percent‚17.06533432006836 proc.cpu.threads‚17$ memory_percent‚96.60451412200928 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 139.76953125( proc.memory.availableMB‚ 556.31640625 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 139.76953125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.in‚0 network.sent‚ 1.0593487e+07 network.recv‚ 1.350263e+06# disk./.usageGB‚1109.02681350708 disk.sda1.out‚05 :‹’þ‚(:‡ ¬¶æÔÞÛÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚97.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î´Í£Fÿ‚(:» ¬¶æÔµÕ†Ú disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0 disk.dm-1.out‚ 140.1796875 network.recv‚ 1.3702e+06 disk.rbd1.out‚ 1.83984375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚84.97705078125 disk.nvme0n1.in‚ 162.94140625# disk.dm-0.out‚14713.51611328125 disk.sda2.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚14713.51611328125 disk.dm-1.in‚ 0.01171875$ memory_percent‚96.60589694976807% disk./.usageGB‚1109.1496849060059 disk.sda3.out‚ 140.1796875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0) proc.memory.percent‚17.06533432006836 cpu‚25.306908560484427 network.sent‚ 1.0709497e+07 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.out‚0( proc.memory.availableMB‚ 556.08984375! proc.memory.rssMB‚ 2795.984375 disk.sda.out‚ 140.1796875 disk.sda1.in‚0 disk.drbd1777.in‚0) disk./.usagePercent‚74.43596599261453 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚14713.51611328125Y«=‘€‚(:† »¶æÔ€ï÷Õ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>,G‚(:¼ »¶æÔœÎé× proc.cpu.threads‚17 network.recv‚ 1.374273e+06% disk./.usageGB‚1109.2738800048828 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.in‚0 disk.rbd2.out‚84.97705078125 disk.drbd1777.out‚0$ memory_percent‚97.08366394042969$ proc.memory.availableMB‚477.8125# proc.memory.rssMB‚ 2874.26171875 cpu‚25.323797498588217( disk./.usagePercent‚74.4443008299068 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 140.76953125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚14714.35302734375& disk.nvme0n1.out‚14714.35302734375 disk.drbd1777.in‚0 network.sent‚ 1.071474e+07 disk.dm-1.out‚ 140.76953125# disk.dm-0.out‚14714.35302734375 disk.sda2.out‚0) proc.memory.percent‚17.54310131072998 disk.sda.in‚ 0.01171875 disk.sda.out‚ 140.76953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0ÐmÔ“‚‚(:ˆ ʶæÔÛŠáÖ gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚157.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H×Û|¥ƒ‚(:¿ ʶæÔóÅÝ disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.54310131072998 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd3.out‚0 cpu‚25.343222000854578 network.sent‚ 1.0727553e+07 disk.sda.out‚ 141.42578125 disk.drbd1777.out‚0 disk.rbd2.out‚85.05517578125% disk./.usageGB‚1109.2733917236328 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.out‚ 141.42578125 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0$ memory_percent‚97.09310531616211( disk./.usagePercent‚74.4442680609424 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚14841.29052734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚14841.29052734375& disk.nvme0n1.out‚14841.29052734375# proc.memory.rssMB‚ 2874.26171875 disk.dm-1.out‚ 141.42578LqÂu¥125 disk.rbd1.out‚ 1.83984375 disk.rbd2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 476.265625 network.recv‚ 1.392162e+06 disk.sda2.in‚0×åЂ“„‚(:ˆ Ù¶æÔ•Ë¼Õ gpu.0.gpu‚57.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚166.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2758 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JõE…‚(:º Ù¶æÔëó©Ù disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 141.984375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚14841.92724609375 cpu‚25.364478475565267 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0 disk.sda3.out‚ 141.984375" disk.nvme0c0n1.in‚ 162.94140625( disk./.usagePercent‚74.4443965767246% disk./.usageGB‚1109.2753067016602 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625# memory_percent‚97.0773458480835 proc.cpu.threads‚17 disk.drbd1777.out‚0# disk.dm-0.out‚14841.92724609375# proc.memory.rssMB‚ 2874.26171875 network.sent‚ 1.0739439e+07 network.recv‚ 1.408718e+06 disk.rbd3.in‚0 disk.sda.out‚ 141.984375 disk.rbd2.out‚86.62939453125& disk.nvme0n1.out‚14841.92724609375( proc.memory.availableMB‚ 478.84765625) proc.memory.percent‚17.54310131072998so::2†- _wandbruntime‚3771  _runtime‚3771ÒÕ*ÁD*‡¾T eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91343005 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91327028 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4037416733790 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884516799990838e+09 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚281362090245120 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚393 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884516799990838e+09 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚281362090245120 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚307239697966080 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚393 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚281398920273920 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 _runtime‚3772.524651503 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversizåÓŸóe_allocations.allocated‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚23179992 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚285451127713280 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0@N¢7R)‹ÌR `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4037289428806 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚285451638546432 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚281398938276864 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4052700269568 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91350307 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚281398920273920 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 _runtime‚3772.524651503 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41068772 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚23179992 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚285451127713280 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚70385624 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91334330 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968Š‘¢zIŒ‚(:’ è¶æÔŽ³ÐÖ gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚163.991' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚32.79820000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êH/H‚(:½ è¶æÔéô¶Ú) proc.memory.percent‚17.54310131072998 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚86.62939453125) disk./.usagePercent‚74.44440016083009 disk.rbd2.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚14842.24365234375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.out‚0# memory_percent‚97.0792293548584# proc.memory.rssMB‚ 2874.26171875 cpu‚25.383587791016897 network.recv‚ 1.420184e+06 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 162.94140625' proc.memory.availableMB‚ 478.5390625 network.sent‚ 1.0749339e+07 disk.sda1.in‚0 disk.sda3.out‚ 142.4140625( disk.nvme0c0n1.out‚14842.24365234375 disk.drbd1777.out‚0 disk.sda.out‚ 142.4140625 disk.rbd0.in‚0& disk.nvme0n1.out‚14842.24365234375% disk./.usageGB‚1109.2753601074219 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 142.4140625̾Ð2Ž- _wandbruntime‚3785  _runtime‚3785ma¡ozKæ¥ :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49772730469703674 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3209821581840515 +jet/accuracy_2_of_2‚0.10337474942207336 particle/accuracy_1_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01822836697101593 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3734525144100189 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3754212558269501 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017625492066144943 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.42307692766189575 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9602470397949219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9824104905128479 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15779593586921692 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941048622131348 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4653462767601013 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6757009029388428 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 particle/accuracy_6_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_4‚0.9110166430473328 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3610779941082001 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24254168570041656 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46527785062789917 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37280702590942383 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5761904716491699 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9773609042167664 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15961310267448425 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08172755688428879 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06798084080219269 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45666247606277466 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20162634551525116 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_4_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46991339325904846 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2266431450843811 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07733957469463348 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1897660791873932 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005949572194367647 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45194685459136963 particle/accuracy_4_of_4‚1 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 jet/accuracy_4_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.322745680809021 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5068826675415039 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886009693145752 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45397913455963135 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46537354588508606 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15432099997997284 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3026057481765747 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5068826675415039 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.42106378078460693 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2894473373889923 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981485068798065 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4334148168563843 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5238288044929504 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9748744368553162 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14800645411014557 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41087567806243896 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_5_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47723862528800964 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 (particle/accuracy‚0.8469794988632202 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28482168912887573 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21293650567531586 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765958786010742 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4199095368385315 jet/accuracy_4_of_6‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878864884376526 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16149042546749115 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996511697769165 particle/accuracy_3_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2233717292547226 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21986007690429688 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12464573979377747 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49374711513519287 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36951348185539246 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5296977162361145 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25557875633239746 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9843401908874512 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16808930039405823 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20527787506580353 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6272894144058228 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5171089768409729 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5270421504974365 ,jet/accuracy_4_of_5‚0.024731183424592018 particle/accuracy_2_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23876093327999115 )loss/val_total_loss‚46.19633362278747 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9864582419395447 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027067702263593674 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.635661780834198 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2550504803657532 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12873855233192444 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4054597318172455 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2797619104385376 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.3710973560810089 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5340971946716309 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.35465115308761597 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05416666716337204 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48016324639320374 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6097158789634705 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48634323477745056 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0276544950902462 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5010439157485962 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5860288739204407 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023343414068222046 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5491468906402588 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3740740716457367 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_4‚1 ,particle/sensitivity‚0.31349602341651917 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06663601845502853 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9646825194358826 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1651688516139984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4880601763725281 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18820159137248993 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 particle/accuracy_7_of_7‚1 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_7‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21785913407802582 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6056131720542908 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6079861521720886 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2918492257595062 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779208898544312 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909926056861877 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166716337204 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Pqê\sGurity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14800645411014557 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14446648955345154 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0876723900437355 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4954971671104431 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5441104769706726 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.033764369785785675 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22672413289546967 +jet/accuracy_3_of_5‚0.08865847438573837 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1570715755224228 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5163894891738892 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5358319282531738 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15056903660297394 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8567526340484619 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1635286808013916 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33646947145462036 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4924437999725342 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17943444848060608 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5921307802200317 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 epoch‚3 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4076598584651947 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5203149914741516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48354119062423706 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8240576982498169 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40185192227363586 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2103448212146759 'particle/f_score‚0.3899940848350525 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016471536830067635 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19842535257339478 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9582054018974304 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9113475680351257 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45828643441200256 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031023046001791954 ,jet/accuracy_3_of_4‚0.055361293256282806 particle/accuracy_6_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16662485897541046 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.551249086856842 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44893139600753784 _step‚395 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864905476570129 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.158299520611763 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960275292396545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09349727630615234 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36934518814086914 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5664284825325012 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3342592120170593 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20586492121219635 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21631333231925964 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9684689044952393 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18070733547210693 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5048581957817078 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9833839535713196 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17624521255493164 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4791334569454193 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26688507199287415 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4942575991153717 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚4835 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9839602708816528 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951009154319763 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129632234573364 jet/accuracy_6_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23805713653564453 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 +particle/specificity‚0.9457070231437683 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008881330490112305 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41335272789001465 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 +jet/accuracy_1_of_1‚0.48283278942108154 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5038076043128967 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01399196032434702 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888887107372284 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40937700867652893 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4125000238418579 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5362160801887512 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13281753659248352 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 _runtime‚3785.407188645 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397586464881897 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18858039379119873 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2962963283061981 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4399781823158264 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1785852015018463 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5895935297012329 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4827724099159241 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.351651668548584 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001976859290152788 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5585317015647888 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4441176652908325 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0034129894338548183 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908420443534851 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1533130258321762 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37103480100631714 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23449045419692993 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20288436114788055 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5243113040924072 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15160906314849854 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884516938508205e+09 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2702702581882477 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4300609529018402 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17193247377872467 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9410919547080994 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48661500215530396 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.055257778614759445 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7484763860702515 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014573877677321434 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5Fɘ&8ë¥ =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4942575991153717 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚4835 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9839602708816528 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951009154319763 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129632234573364 jet/accuracy_6_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23805713653564453 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 +particle/specificity‚0.9457070231437683 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008881330490112305 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41335272789001465 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5445402264595032 +jet/accuracy_1_of_1‚0.48283278942108154 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5038076043128967 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01399196032434702 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888887107372284 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40937700867652893 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4125000238418579 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5362160801887512 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13281753659248352 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 _runtime‚3785.407188645 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397586464881897 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18858039379119873 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2962963283061981 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4399781823158264 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1785852015018463 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5895935297012329 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4827724099159241 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.351651668548584 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001976859290152788 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5585317015647888 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4441176652908325 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0034129894338548183 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908420443534851 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1533130258321762 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37103480100631714 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23449045419692993 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20288436114788055 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5243113040924072 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15160906314849854 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884516938508205e+09 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2702702581882477 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4300609529018402 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17193247377872467 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9410919547080994 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48661500215530396 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.055257778614759445 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7484763860702515 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014573877677321434 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49772730469703674 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3209821581840515 +jet/accuracy_2_of_2‚0.10337474942207336 particle/accuracy_1_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01822836697101593 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3734525144100189 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3754212558269501 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017625492066144943 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.42307692766189575 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9602470397949219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9824104905128479 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15779593586921692 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941048622131348 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4653462767601013 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6757009029388428 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 particle/accuracy_6_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_4‚0.9110166430473328 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3610779941082001 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24254168570041656 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.46527785062789917 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37280702590942383 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5761904716491699 jet/accuracy_3_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9773609042167664 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15961310267448425 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08172755688428879 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06798084080219269 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.45666247606277466 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20162634551525116 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_4_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46991339325904846 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 PuвÖsZrity/2FRt0SRqqt*FBt/FRt_purity‚0.2266431450843811 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07733957469463348 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1897660791873932 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005949572194367647 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45194685459136963 particle/accuracy_4_of_4‚1 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 jet/accuracy_4_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.322745680809021 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5068826675415039 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886009693145752 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45397913455963135 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46537354588508606 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15432099997997284 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3026057481765747 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5068826675415039 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_5‚0.42106378078460693 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2894473373889923 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981485068798065 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4334148168563843 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5238288044929504 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9748744368553162 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14800645411014557 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.41087567806243896 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_5_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47723862528800964 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 (particle/accuracy‚0.8469794988632202 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28482168912887573 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21293650567531586 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765958786010742 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4199095368385315 jet/accuracy_4_of_6‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878864884376526 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16149042546749115 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996511697769165 particle/accuracy_3_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2233717292547226 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21986007690429688 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12464573979377747 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49374711513519287 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36951348185539246 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5296977162361145 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25557875633239746 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9843401908874512 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16808930039405823 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20527787506580353 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6272894144058228 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5171089768409729 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5270421504974365 ,jet/accuracy_4_of_5‚0.024731183424592018 particle/accuracy_2_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23876093327999115 )loss/val_total_loss‚46.19633362278747 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9864582419395447 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027067702263593674 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.635661780834198 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2550504803657532 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12873855233192444 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4054597318172455 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2797619104385376 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.3710973560810089 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5340971946716309 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.35465115308761597 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05416666716337204 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48016324639320374 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6097158789634705 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48634323477745056 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0276544950902462 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5010439157485962 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5860288739204407 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023343414068222046 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5491468906402588 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3740740716457367 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_4‚1 ,particle/sensitivity‚0.31349602341651917 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06663601845502853 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9646825194358826 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1651688516139984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4880601763725281 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18820159137248993 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 particle/accuracy_7_of_7‚1 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_7‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21785913407802582 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6056131720542908 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6079861521720886 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2918492257595062 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9779208898544312 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909926056861877 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166716337204 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14800645411014557 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14446648955345154 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0876723900437355 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4954971671104431 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5441104769706726 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.033764369785785675 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22672413289546967 +jet/accuracy_3_of_5‚0.08865847438573837 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1570715755224228 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5163894891738892 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5358319282531738 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15056903660297394 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8567526340484619 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1635286808013916 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33646947145462036 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4924437999725342 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17943444848060608 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5921307802200317 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 epoch‚3 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4076598584651947 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5203149914741516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48354119062423706 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8240576982498169 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40185192227363586 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2103448212146759 'particle/f_score‚0.3899940848350525 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016471536830067635 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19842535257339478 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9582054018974304 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9113475680351257 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45828643441200256 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031023046001791954 ,jet/accuracy_3_of_4‚0.055361293256282806 particle/accuracy_6_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16662485897541046 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.551249086856842 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44893139600753784 _step‚395 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864905476570129 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.158299520611763 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960275292396545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09349727630615234 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36934518814086914 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5664284825325012 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3342592120170593 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20586492121219635 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21631333231925964 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9684689044952393 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18070733547210693 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5048581957817078 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9833839535713196 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17624521255493164 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4791334569454193 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26688507199287415 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109‹x‡µÇ‘‚(:’ ÷¶æÔ©ŽÛÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.315399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÓ¶”G’‚(:¼ ÷¶æÔ‡³ÖØ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2873.82421875 disk.nvme0n1.in‚ 162.94140625 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0) disk./.usagePercent‚74.44505682015541 disk.sda2.in‚0 disk.rbd2.out‚86.70751953125 disk.sda1.in‚0* proc.memory.percent‚17.540431022644043 cpu‚25.38740644750359 network.recv‚ 1.426272e+06 disk.dm-0.in‚ 162.94140625! disk.dm-0.out‚15323.896484375 disk.dm-1.out‚ 142.96484375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0c0n1.out‚15323.896484375 disk.rbd1.out‚ 1.83984375 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# memory_percent‚91.2893533706665$ disk.nvme0n1.out‚15323.896484375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 142.96484375 disk.sda1.out‚0) proc.memory.availableMB‚ 1427.15234375 network.sent‚ 1.0857096e+07% disk./.usageGB‚1109.2851448059082 disk.sda3.out‚ 142.96484375,Ub£2“- _wandbruntime‚3799  _runtime‚3799Úbˑɔà _runtime‚3799.464823753 _step‚396 ' lr-AdamW/pg1‚0.00013791344372462142 ' lr-AdamW/pg2‚0.00013791344372462142 trainer/global_step‚4849 % _timestamp‚1.7884517079104092e+09ñ ’Î•È ' lr-AdamW/pg2‚0.00013791344372462142 trainer/global_step‚4849 % _timestamp‚1.7884517079104092e+09 _runtime‚3799.464823753 _step‚396 ' lr-AdamW/pg1‚0.00013791344372462142Œ°ÌÉ|2–- _wandbruntime‚3799  _runtime‚3799bJ~v—×O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286678426648064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4068108491776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4067452129280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308571417824768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚3799.465817034 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4067452129280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 trainer/global_step‚4849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23296778 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4053128269008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286678426648064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.smallò?,Ôg _pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4068108491776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275194743995392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884517079111848e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91669310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279247889303760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91669310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275194761034752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048^-â'˜ÜO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64556736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279247217826644 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41259510 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286677752282624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4067452129280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 trainer/global_step‚4849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23296778 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4053128269008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286678426648064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4068108491776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275194743995392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 % _timestamp‚1.7884517079111848e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91669310 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279247889303760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91669310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275194761034752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286678426648064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4068108491776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4067452129280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308571417824768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚3799.465817034 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279302938392168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308633068562432 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286735599128064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266894360064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4068927932928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70722324 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23301277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884517086083553e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521ŒKñW{&›æM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚3800.163233698 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308633068562432 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286735599128064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266894360064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4068927932928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70722324 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91707828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23301277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286734924749312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884517086083553e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70722326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275249648125952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279302938392168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91687764 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289537064396288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71409110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92595362 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289536390030848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093963608416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164004470 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496ÃGâ'¨ÜO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41667908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164004470 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71409108 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23523122 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108437935104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71409110 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285427970098688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311653930767872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884517444672208e+09 _runtime‚3836.021818942 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285427952095744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315963254932992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚4899 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282033931171168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4093309170660 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108437935104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311653954707968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23523549 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289537064396288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71409110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92595362 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289536390030848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65191459 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093963608416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872‘ßÐʵ2©- _wandbruntime‚3836  _runtime‚3836Sã㇪àM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4899 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289594236876288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788451745162777e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _runtime‚3836.718121331 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285484323137536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71423126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_b„"_atch_end/active_bytes.small_pool.allocated‚ 4109913738752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316025574136320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808¢Æµ“ë&«åM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316025574136320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311715581505536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109913738752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚4899 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289594236876288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788451745162777e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _runtime‚3836.718121331 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285484323137536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71423126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23527545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92613816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430’CIaj2¬- _wandbruntime‚3836  _runtime‚3836Bˆý‰`­Ú 0loss/FRt4/assignment_loss‚4.2649899952941475 0loss/FBt4/assignment_loss‚0.7285144900535949 /loss/FRt4/detection_loss‚2.8463979131645627 &loss/total_loss‚46.150947634573534 trainer/global_step‚4899 _step‚403 0loss/FRt1/assignment_loss‚3.1557201702486384 0loss/FRt2/assignment_loss‚1.4958225344166611 1loss/FBt3/assignment_loss‚0.03400266477838158 /loss/FBt2/detection_loss‚0.6476866039632562 /loss/FRt2/detection_loss‚1.5209510466366103 /loss/FBt1/detection_loss‚3.5691192150115967 epoch‚4 % _timestamp‚1.7884517451651216e+09 _runtime‚3836.719467093 0loss/FRt3/assignment_loss‚3.2693158194076184 1loss/SRqqt1/detection_loss‚1.8360226574540137 2loss/SRqqt4/assignment_loss‚2.1237586992127553 0loss/FBt1/assignment_loss‚0.4787059894626639 1loss/SRqqt2/detection_loss‚0.9499154455740663 /loss/FBt4/detection_loss‚1.9475278074924762 2loss/SRqqt2/assignment_loss‚1.1408424453316803 1loss/FBt2/assignment_loss‚0.17671817102365597 1loss/SRqqt3/detection_loss‚1.7666365896853118 2loss/SRqqt3/assignment_loss‚1.4784822487630522 2loss/SRqqt1/assignment_loss‚1.5133041175268591 /loss/FRt1/detection_loss‚2.4249023788354616 /loss/FRt3/detection_loss‚2.3339657007261763 0loss/SRqqt4/detection_loss‚2.931565862326395 .loss/FBt3/detection_loss‚3.516079068183899A}ÚKe®ß 1loss/FBt2/assignment_loss‚0.17671817102365597 1loss/SRqqt3/detection_loss‚1.7666365896853118 2loss/SRqqt3/assignment_loss‚1.4784822487630522 2loss/SRqqt1/assignment_loss‚1.5133041175268591 /loss/FRt1/detection_loss‚2.4249023788354616 /loss/FRt3/detection_loss‚2.3339657007261763 0loss/SRqqt4/detection_loss‚2.931565862326395 .loss/FBt3/detection_loss‚3.516079068183899 0loss/FRt4/assignment_loss‚4.2649899952941475 0loss/FBt4/assignment_loss‚0.7285144900535949 /loss/FRt4/detection_loss‚2.8463979131645627 &loss/total_loss‚46.150947634573534 trainer/global_step‚4899 _step‚403 0loss/FRt1/assignment_loss‚3.1557201702486384 0loss/FRt2/assignment_loss‚1.4958225344166611 1loss/FBt3/assignment_loss‚0.03400266477838158 /loss/FBt2/detection_loss‚0.6476866039632562 /loss/FRt2/detection_loss‚1.5209510466366103 /loss/FBt1/detection_loss‚3.5691192150115967 epoch‚4 % _timestamp‚1.7884517451651216e+09 _runtime‚3836.719467093 0loss/FRt3/assignment_loss‚3.2693158194076184 1loss/SRqqt1/detection_loss‚1.8360226574540137 2loss/SRqqt4/assignment_loss‚2.1237586992127553 0loss/FBt1/assignment_loss‚0.4787059894626639 1loss/SRqqt2/detection_loss‚0.9499154455740663 /loss/FBt4/detection_loss‚1.9475278074924762 2loss/SRqqt2/assignment_loss‚1.1408424453316803“¶yƒ©“¯‚(:ˆ ¤·æÔï»ÏÕ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ÂÓL°‚(:Á ¤·æÔñÊ•Ø disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2873.82421875 cpu‚25.383485613987773 disk.sda2.in‚0 disk.sda3.out‚ 144.33984375 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 144.33984375 disk.rbd2.in‚0* proc.memory.percent‚17.540431022644043# disk.dm-0.out‚15798.69189453125 disk.dm-1.out‚ 144.33984375 disk.rbd1.out‚ 1.83984375( disk.nvme0c0n1.out‚15798.69189453125 network.sent‚ 1.1095191e+07 network.recv‚ 1.442338e+06) disk./.usagePercent‚74.44493188847869% disk./.usageGB‚1109.2832832336426 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875# memory_percent‚91.2930965423584 disk.dm-0.in‚ 162.94140625 disk.sda2.out‚0& disk.nvme0n1.out‚15798.69189453125 disk.drbd1777.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 162.94140625( proc.memory.availableMB‚ 1426.5390625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd2.out‚ 86.8173828125ýû‰’±‚(:‡ ³·æÔ¹€óÕ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚164.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.811 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÂñK²‚(:À ³·æÔ«Ÿ­Ú disk.sda3.out‚ 144.88671875* proc.memory.percent‚17.540431022644043 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚15896.40283203125 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375# disk./.usageGB‚1109.40824508667 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 144.88671875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 144.88671875 disk.sda.in‚ 0.01171875) disk./.usagePercent‚74.45331818328533" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0$ memory_percent‚91.93778038024902 proc.cpu.threads‚17( proc.memory.availableMB‚ 1320.9140625# proc.memory.rssMB‚ 2873.82421875 cpu‚25.382166795318877 network.sent‚ 1.1205194e+07& disk.nvme0n1.out‚15896.40283203125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚15896.40283203125 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 network.recv‚ 1.448098e+06 disk.rbd2.out‚ 88.4775390625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0q»DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314736491591168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4352217192960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4352374525440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93521414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 trainer/global_step‚4949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288245622041088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93541452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÙbÎVκÜO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocatiíîì&on.large_pool.allocated‚72109910 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288245604038144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚405 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134798947824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3873.841115154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65826755 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23750882 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150080103424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284819301561460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42076310 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292395027779072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288245604038144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314736491591168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4352217192960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4352374525440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93521414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 trainer/global_step‚4949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288245622041088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280685174090752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93541452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0•Ú‚áº2»- _wandbruntime‚3874  _runtime‚3874|–Øîç&¼áM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72123924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3874.559012854 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288301975079936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284875693604308 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 _step‚406 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4150899544576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4150243168768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4353069066240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4150899544576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23755013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42084476 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884517830033026e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165663792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93559932 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414@t{Þì&½æM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42084476 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884517830033026e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165663792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93559932 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292452874624512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72123924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3874.559012854 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288301975079936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284875693604308 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314798142328832 _step‚406 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4150899544576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292452874624512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4150243168768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4353069066240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4135615382996 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4150899544576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23755450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23755013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844–FJe2¾- _wandbruntime‚3874  _runtime‚38746°¸º_¿Ù 1loss/FBt1/assignment_loss‚0.16585533188238819 .loss/FRt4/detection_loss‚3.572961765527725 /loss/FBt2/detection_loss‚0.7845606006855841 % _timestamp‚1.7884517830058322e+09 0loss/FRt4/assignment_loss‚4.5661357879638675 2loss/SRqqt2/assignment_loss‚1.0087001147124195 2loss/SRqqt3/assignment_loss‚1.5431877047969744 /loss/FRt2/detection_loss‚2.0787318421770697 0loss/SRqqt2/detection_loss‚1.183847721906439 &loss/total_loss‚47.776738625039364 epoch‚4 1loss/SRqqt3/detection_loss‚1.7435950166904008 1loss/SRqqt4/detection_loss‚2.8745336566980066 _step‚407 0loss/FBt3/assignment_loss‚0.3008537569216157 1loss/FBt2/assignment_loss‚0.26903836013786114 1loss/SRqqt1/detection_loss‚1.7833515112682925 /loss/FBt1/detection_loss‚2.9337262908617654 0loss/FRt1/assignment_loss‚3.1733140704723506 1loss/SRqqt4/assignment_loss‚2.185576014793836 0loss/FBt4/assignment_loss‚0.5265003974491265 /loss/FRt3/detection_loss‚2.0295171755552293 /loss/FBt4/detection_loss‚3.4416244540895735 trainer/global_step‚4949 _runtime‚3874.560161136 0loss/FRt2/assignment_loss‚1.6350058498589888 2loss/SRqqt1/assignment_loss‚1.5430646074651662 /loss/FRt1/detection_loss‚2.2184719993517947 /loss/FBt3/detection_loss‚2.9131142497062683 0loss/FRt3/assignment_loss‚3.3014703440666198u³[dÀÞ .loss/FRt4/detection_loss‚3.572961765527725 /loss/FBt2/detection_loss‚0.7845606006855841 % _timestamp‚1.7884517830058322e+09 0loss/FRt4/assignment_loss‚4.5661357879638675 2loss/SRqqt2/assignment_loss‚1.0087001147124195 2loss/SRqqt3/assignment_loss‚1.5431877047969744 /loss/FRt2/detection_loss‚2.0787318421770697 0loss/SRqqt2/detection_loss‚1.183847721906439 &loss/total_loss‚47.776738625039364 epoch‚4 1loss/SRqqt3/detection_loss‚1.7435950166904008 1loss/SRqqt4/detection_loss‚2.8745336566980066 _step‚407 0loss/FBt3/assignment_loss‚0.3008537569216157 1loss/FBt2/assignment_loss‚0.26903836013786114 1loss/SRqqt1/detection_loss‚1.7833515112682925 /loss/FBt1/detection_loss‚2.9337262908617654 0loss/FRt1/assignment_loss‚3.1733140704723506 1loss/SRqqt4/assignment_loss‚2.185576014793836 0loss/FBt4/assignment_loss‚0.5265003974491265 /loss/FRt3/detection_loss‚2.0295171755552293 /loss/FBt4/detection_loss‚3.4416244540895735 trainer/global_step‚4949 _runtime‚3874.560161136 0loss/FRt2/assignment_loss‚1.6350058498589888 2loss/SRqqt1/assignment_loss‚1.5430646074651662 /loss/FRt1/detection_loss‚2.2184719993517947 /loss/FBt3/detection_loss‚2.9131142497062683 0loss/FRt3/assignment_loss‚3.3014703440666198 1loss/FBt1/assignment_loss‚0.16585533188238819——¤³ºÁ‚(:ˆ Ñ·æÔž¾äÕ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.¶ÛÐÙpowerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MM˜§I‚(:¾ Ñ·æÔÙåàØ disk.sda3.out‚ 145.8984375 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.sda1.out‚0# disk.dm-0.out‚15997.81591796875 disk.rbd1.out‚ 1.83984375 disk.drbd1777.out‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚15997.81591796875 network.recv‚ 1.456621e+06) disk./.usagePercent‚74.46190262792348( disk.nvme0c0n1.out‚15997.81591796875( proc.memory.availableMB‚ 950.42578125* proc.memory.percent‚17.540431022644043 disk.rbd2.out‚ 88.5556640625 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2873.82421875 proc.cpu.threads‚17 disk.sda.out‚ 145.8984375 disk.nvme0n1.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.dm-1.out‚ 145.8984375 network.sent‚ 1.1214849e+07% disk./.usageGB‚1109.5361595153809 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 cpu‚25.379640185489333 disk.rbd3.out‚0# memory_percent‚94.1990613937378tgñžÃ‚(:“ à·æÔË̆Ö gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.486' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.097199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÕ+”EÄ‚(:º à·æÔ‡»ŒÙ disk.sda.in‚ 0.01171875 disk.rbd0.in‚0# disk.dm-0.out‚15998.15966796875 network.recv‚ 1.463068e+06 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 146.3359375 disk.rbd3.out‚0 disk.sda3.out‚ 146.3359375 disk.rbd2.out‚ 88.5556640625" disk.nvme0c0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625$ proc.memory.availableMB‚950.4375 cpu‚25.37837614350079 disk.sda2.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd0.out‚0& disk.nvme0n1.out‚15998.15966796875 disk.drbd1777.in‚0 network.sent‚ 1.1323663e+07 disk.dm-1.in‚ 0.01171875 disk.rbd2.in‚0( disk.nvme0c0n1.out‚15998.15966796875 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2873.83203125* proc.memory.percent‚17.540478706359863 disk.sda.out‚ 146.3359375 disk.rbd1.in‚0$ memory_percent‚94.19898986816406) disk./.usagePercent‚74.47881090152991% disk./.usageGB‚1109.7881050109863ŠŒ$(2Å- _wandbruntime‚3912  _runtime‚3912¹•"ÉÆà ' lr-AdamW/pg1‚0.00013440824123823054 ' lr-AdamW/pg2‚0.00013440824123823054 trainer/global_step‚4999 % _timestamp‚1.7884518208328528e+09 _runtime‚3912.387236891 _step‚408š½ÎÇÈ _step‚408 ' lr-AdamW/pg1‚0.00013440824123823054 ' lr-AdamW/pg2‚0.00013440824123823054 trainer/global_step‚4999 % _timestamp‚1.7884518208328528e+09 _runtime‚3912.387236891˜xhV2È- _wandbruntime‚3912  _runtime‚3912»vj Ý'É×O =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884518208336537e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94447466 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317819004534272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚409 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395550673408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94447466 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1;›hâ'ÊÜO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395550673408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42484708 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94447466 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884518208336537e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94447466 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317819004534272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚409™¬‘2Ë- _wandbruntime‚3913  _runtime‚3913iýD»"ÌàM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 _step‚410 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295310837993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94465920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287661063994392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317880679212032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167310710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23983069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283485267709952 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚3913.105797946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23983526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396257382400©·¦ë&ÍåM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4191228974592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291119627022336 $ _timestamp‚1.788451821550442e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚4999 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291119609019392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317880679212032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167310710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23983069 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283485267709952 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚3913.105797946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72824724 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23983526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322276912654336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396257382400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4175796284440 _step‚410 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295310837993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94465920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287661063994392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95373518 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298112303275520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24206837 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚3950.452829245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168885026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231395352576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788451858898106e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95393556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414ÊE ²ÜÛO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293880907922944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325340098780160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95373518 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298112303275520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _runtime‚3950.452829245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168885026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95373518 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231395352576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788451858898106e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95393556 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95393556 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231395352576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73511508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚5049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298112977640960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293880907922944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95391972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4232871156224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73525524 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42901278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4232871156224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286230491277312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67113131 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 trainer/global_step‚5049 _runtime‚3951.153738599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298169475742208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884518595985405e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325402797762560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414„›_ ì&ßæM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286230491277312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320963192155136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67113131 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 trainer/global_step‚5049 _runtime‚3951.153738599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298169475742208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884518595985405e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325402797762560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298170150120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95391972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4232871156224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73525524 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42901278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4232871156224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288920793674752 trainer/global_step‚5099 % _timestamp‚1.7884518962178147e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328465818456064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296698577868288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300971615389184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048‘Q$3â'ìÜO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296698577868288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300971615389184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3987.772480674 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4273037520896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4481716215296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296698559865344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288920793674752 trainer/global_step‚5099 % _timestamp‚1.7884518962178147e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328465818456064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¡®'ç¿2í- _wandbruntime‚3988  _runtime‚3988Ê»(ç&îáM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4273200586240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884518969207518e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4258121401220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74226324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296754912904192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4273856962048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4273200586240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328528331157504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3988.475850689 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170564414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74226326 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324045752978432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FïæM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43309676 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884518969207518e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C=DeviceStatsMonitor.on_train_batch_end/oversize_seg½h¨®ments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296754930907136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4258121401220 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74226324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296754912904192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4273856962048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4273200586240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74226326 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328528331157504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96338088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚3988.475850689 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170564414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74226326 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324045752978432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97225622 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884519337338424e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24663679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913108 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚5149 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43709908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97225622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4313366964224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0õ!v‚þÜO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4313366964224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913108 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚5149 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68373589 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43709908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913108 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97225622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4313366964224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291665983163392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172138730 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_sta¯x`’` rt/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97225622 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884519337338424e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24663679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944¥}àt[2ÿ- _wandbruntime‚4025  _runtime‚4025© Yç&€áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _runtime‚4025.983068299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303887425617408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291720887293952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5149 _step‚422 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327128289861632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296019189596616 % _timestamp‚1.7884519344280457e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚4141YJì&æM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74927126 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4525635899904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74927124 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303887425617408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68386291 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97264140 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299572564846592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296019189596616 % _timestamp‚1.7884519344280457e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _runtime‚4025.983068299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303887425617408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291720887293952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5149 _step‚422 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24667771 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327128289861632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327128265921536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¦»ëÞ2‚- _wandbruntime‚4025  _runtime‚4025PÝËXƒÒ .loss/FRt1/detection_loss‚2.132274327799678 0loss/SRqqt1/detection_loss‚1.688987336307764 trainer/global_step‚5149 0loss/FBt4/assignment_loss‚0.3870108150903517 /loss/FRt2/detection_loss‚1.6888003800330418 /loss/FRt4/detection_loss‚2.8975115212900886 1loss/SRqqt2/detection_loss‚1.2868675660075886 /loss/FBt2/detection_loss‚0.8368782355316559 /loss/FBt3/detection_loss‚3.0107727213339373 0loss/FBt1/assignment_loss‚0.3190840639373554 1loss/SRqqt3/detection_loss‚1.6347641141006821 .loss/FBt4/detection_loss‚2.683861504901539 /loss/FRt1/assignment_loss‚2.729965856919686 0loss/FRt2/assignment_loss‚1.7355851754546165 /loss/FRt4/assignment_loss‚4.037825029471825 2loss/SRqqt3/assignment_loss‚1.1748745488493066 -loss/FRt3/detection_loss‚1.99118770821528 %loss/total_loss‚42.42484652947383 epoch‚4 2loss/SRqqt4/assignment_loss‚1.7137627789267786 0loss/FBt3/assignment_loss‚0.2610340223705779 0loss/SRqqt4/detection_loss‚2.610686327572222 _runtime‚4025.984104039 0loss/FRt3/assignment_loss‚2.5564316955479707 2loss/SRqqt2/assignment_loss‚0.7912011507100293 1loss/FBt2/assignment_loss‚0.17911203222489283 /loss/FBt1/detection_loss‚2.6593382358551025 2loss/SRqqt1/assignment_loss‚1.4170293810218573 % _timestamp‚1.7884519344297984e+09 _step‚423¸jª…]„× 0loss/FRt2/assignment_loss‚1.7355851754546165 /loss/FRt4/assignment_loss‚4.037825029471825 2loss/SRqqt3/assignment_loss‚1.1748745488493066 -loss/FRt3/detection_loss‚1.99118770821528 %loss/total_loss‚42.42484652947383 epoch‚4 2loss/SRqqt4/assignment_loss‚1.7137627789267786 0loss/FBt3/assignment_loss‚0.2610340223705779 0loss/SRqqt4/detection_loss‚2.610686327572222 _runtime‚4025.984104039 0loss/FRt3/assignment_loss‚2.5564316955479707 2loss/SRqqt2/assignment_loss‚0.7912011507100293 1loss/FBt2/assignment_loss‚0.17911203222489283 /loss/FBt1/detection_loss‚2.6593382358551025 2loss/SRqqt1/assignment_loss‚1.4170293810218573 % _timestamp‚1.7884519344297984e+09 _step‚423 .loss/FRt1/detection_loss‚2.132274327799678 0loss/SRqqt1/detection_loss‚1.688987336307764 trainer/global_step‚5149 0loss/FBt4/assignment_loss‚0.3870108150903517 /loss/FRt2/detection_loss‚1.6888003800330418 /loss/FRt4/detection_loss‚2.8975115212900886 1loss/SRqqt2/detection_loss‚1.2868675660075886 /loss/FBt2/detection_loss‚0.8368782355316559 /loss/FBt3/detection_loss‚3.0107727213339373 0loss/FBt1/assignment_loss‚0.3190840639373554 1loss/SRqqt3/detection_loss‚1.6347641141006821 .loss/FBt4/detection_loss‚2.683861504901539 /loss/FRt1/assignment_loss‚2.729965856919686§êú…F‘…‚(:† ç¸æÔÜö× gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.755' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.151 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú¹ÃäN†‚(:à ç¸æÔ¼äœØ disk.drbd1777.out‚0* proc.memory.percent‚17.540478706359863 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 151.31640625 disk.sda3.out‚ 151.31640625 disk.drbd1777.in‚0 cpu‚25.36759332216554 disk.dm-0.in‚ 162.94140625 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2873.83203125% disk./.usageGB‚1110.2875137329102# disk.dm-0.out‚16585.83056640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 162.94140625$ memory_percent‚94.19980049133301' proc.memory.availableMB‚ 950.3046875 proc.cpu.threads‚17 network.sent‚ 1.1682629e+07 disk.rbd0.out‚0) disk./.usagePercent‚74.51232663989006 disk.dm-1.out‚ 151.31640625 disk.rbd3.out‚0& disk.nvme0n1.out‚16585.83056640625 network.recv‚ 1.50522e+06 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 2.08349609375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚92.29150390625( disk.nvme0c0n1.out‚16585.83056640625ðf·ç‘‡‚(:† ö¸æÔÅÀÆÕ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚90.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.043 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ ÞBKˆ‚(:À ö¸æÔÆžœÙ" disk.nvme0c0n1.in‚ 162.94140625 cpu‚25.366350528284947 disk.drbd1777.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0' disk.nvme0c0n1.out‚16683.3330078125 disk.nvme0n1.in‚ 162.94140625$ memory_percent‚94.19982433319092* proc.memory.percent‚17.540478706359863 network.sent‚ 1.1791393e+07 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 network.recv‚ 1.511629e+06 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 151.7578125# proc.memory.rssMB‚ 2873.83203125) disk./.usagePercent‚74.51232279977704 disk.rbd0.out‚0% disk.nvme0n1.out‚16683.3330078125" disk.dm-0.out‚16683.3330078125 disk.rbd1.out‚ 2.08349609375 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.out‚ 151.7578125 disk.sda1.out‚0 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 950.30078125 proc.cpu.threads‚17% disk./.usageGB‚1110.2874565124512 disk.sda.out‚ 151.7578125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚92.29150390625^Å<2‰- _wandbruntime‚4062  _runtime‚4062n^ ÉŠÃ ' lr-AdamW/pg2‚0.00012961249829204698 trainer/global_step‚5199 % _timestamp‚1.7884519707803795e+09 _runtime‚4062.334817711 _step‚424 ' lr-AdamW/pg1‚0.00012961249829204698üš8|Î‹È ' lr-AdamW/pg2‚0.00012961249829204698 trainer/global_step‚5199 % _timestamp‚1.7884519707803795e+09 _runtime‚4062.334817711 _step‚424 ' lr-AdamW/pg1‚0.00012961249829204698¨šnã2Œ- _wandbruntime‚4062  _runtime‚4062ë=PŽ ×O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355009132544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294411189691392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884519707811787e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚425 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 TGDeviceStatsMonitor.on_train_batch_ståó¾åOart/allocation.large_pool.allocated‚75613910 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98151674 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69010961 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302333863750144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338321207108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088¹ »vâ'ŽÜO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302333863750144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355009132544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294411206730752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294411189691392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884519707811787e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚425 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75613910 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98151674 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69010961 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302333863750144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338321207108 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430©fæ—»2- _wandbruntime‚4063  _runtime‚4063V£æ&àM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788451971743478e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302390216788992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294466093821952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4355828573696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092ãB^=”‘åM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24896649 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4355828573696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98190192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294466093821952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98170128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294466110861312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NA½PWDeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306745388986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4355828573696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24897092 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788451971743478e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69023570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302390216788992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Pˆ>š‚(:³ £¹æÔëç‰Ù disk.rbd3.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚16781.9521484375 disk.sda2.in‚0' proc.memory.availableMB‚ 950.0859375 proc.cpu.threads‚17 network.recv‚ 1.525847e+06% disk./.usageGB‚1110.2875137329102 disk.sda.in‚ 0.01171875 disk.sda.out‚153.5625 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375* proc.memory.percent‚17.540478706359863) disk./.usagePercent‚74.51232663989006" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 94.1748046875 disk.sda2.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚16781.9521484375 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2873.83203125 disk.sda3.out‚153.5625 disk.rbd0.out‚0% disk.nvme0n1.out‚16781.9521484375 cpu‚25.362958123254433 network.sent‚ 1.1910874e+07 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.sda3.in‚ 0.01171875 disk.dm-1.out‚153.5625$ memory_percent‚94.20113563537598S<öó2›- _wandbruntime‚4100  _runtime‚4100O¹ÊȜ ' lr-AdamW/pg2‚0.00012839341170500754 trainer/global_step‚5249 $ _timestamp‚1.788452008531353e+09 _runtime‚4100.085737524 _step‚428 ' lr-AdamW/pg1‚0.00012839341170500754 žçÍÇ ' lr-AdamW/pg1‚0.00012839341170500754 ' lr-AdamW/pg2‚0.00012839341170500754 trainer/global_step‚5249 $ _timestamp‚1.788452008531353e+09 _runtime‚4100.085737524 _step‚428¬Ï¥],2ž- _wandbruntime‚4100  _runtime‚4100ío÷ÉÜ'ŸÖO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69647628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379810984272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309547528633856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76314708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25121369 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333231712890368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305151515692544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76314710 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99097764 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077726 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76314710 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99097764 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99077726 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305151515692544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4396158003712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚297211300349952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175444970 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25125420 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99096180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309604701113856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333293339687936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305207886734336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76328724 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337905091090944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301591944808756 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99096180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44534878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4611751403008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432)©Ñ£é&£ãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175444970 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25125420 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99096180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309604701113856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309604701113856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333293339687936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305207886734336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305207868731392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76328724 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337905091090944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301591944808756 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99096180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44534878 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305207886734336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76328726 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4611751403008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301591273331432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76328724 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4396158003712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚297211300349952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928®pòÊ:2¤- _wandbruntime‚4100  _runtime‚4100z¢áX¥Ò 2loss/SRqqt2/assignment_loss‚1.0659813613338924 0loss/FBt1/assignment_loss‚0.2916178712621331 _step‚431 2loss/SRqqt3/assignment_loss‚1.7823382243350672 1loss/FBt2/assignment_loss‚0.23002297850854347 0loss/SRqqt3/detection_loss‚1.646775081230883 .loss/FBt3/detection_loss‚2.138817052046458 %loss/total_loss‚45.11762819934071 _runtime‚ 4100.80628959 /loss/FRt3/assignment_loss‚3.584070857275616 trainer/global_step‚5249 /loss/FRt4/assignment_loss‚4.374642959347478 1loss/FBt4/assignment_loss‚0.36816163262765006 1loss/SRqqt1/detection_loss‚1.8337859626935453 .loss/FBt2/detection_loss‚0.718441539339783 2loss/SRqqt4/assignment_loss‚1.9354645322109092 /loss/FRt2/detection_loss‚1.7178442460649155 /loss/FRt4/detection_loss‚2.9665749382089683 epoch‚4 2loss/SRqqt1/assignment_loss‚1.8891519053734611 /loss/FBt3/assignment_loss‚0.523715008671085 .loss/FRt3/detection_loss‚2.185685770078139 1loss/SRqqt2/detection_loss‚1.1397313756951646 .loss/FBt4/detection_loss‚2.993918697039286 0loss/FRt2/assignment_loss‚1.7334573093582601 /loss/FRt1/detection_loss‚2.1567916901821786 1loss/SRqqt4/detection_loss‚2.5317663920336755 /loss/FBt1/detection_loss‚2.5695815458893776 % _timestamp‚1.7884520092518873e+09 0loss/FRt1/assignment_loss‚2.7392892685342343¬Š]¦× /loss/FRt2/detection_loss‚1.7178442460649155 /loss/FRt4/detection_loss‚2.9665749382089683 epoch‚4 2loss/SRqqt1/assignment_loss‚1.8891519053734611 /loss/FBt3/assignment_loss‚0.523715008671085 .loss/FRt3/detection_loss‚2.185685770078139 1loss/SRqqt2/detection_loss‚1.1397313756951646 .loss/FBt4/detection_loss‚2.993918697039286 0loss/FRt2/assignment_loss‚1.7334573093582601 /loss/FRt1/detection_loss‚2.1567916901821786 1loss/SRqqt4/detection_loss‚2.5317663920336755 /loss/FBt1/detection_loss‚2.5695815458893776 % _timestamp‚1.7884520092518873e+09 0loss/FRt1/assignment_loss‚2.7392892685342343 2loss/SRqqt2/assignment_loss‚1.0659813613338924 0loss/FBt1/assignment_loss‚0.2916178712621331 _step‚431 2loss/SRqqt3/assignment_loss‚1.7823382243350672 1loss/FBt2/assignment_loss‚0.23002297850854347 0loss/SRqqt3/detection_loss‚1.646775081230883 .loss/FBt3/detection_loss‚2.138817052046458 %loss/total_loss‚45.11762819934071 _runtime‚ 4100.80628959 /loss/FRt3/assignment_loss‚3.584070857275616 trainer/global_step‚5249 /loss/FRt4/assignment_loss‚4.374642959347478 1loss/FBt4/assignment_loss‚0.36816163262765006 1loss/SRqqt1/detection_loss‚1.8337859626935453 .loss/FBt2/detection_loss‚0.718441539339783 2loss/SRqqt4/assignment_loss‚1.9354645322109092¯]õ“§‚(:ˆ ²¹æÔ¢…éÖ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–VèM¨‚(: ²¹æÔŒ°á×) disk./.usagePercent‚74.51233099201814 disk.drbd1777.in‚0# disk.dm-0.out‚16879.62841796875 cpu‚25.361839756068456" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚16879.62841796875& disk.nvme0n1.out‚16879.62841796875 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875* proc.memory.percent‚17.540478706359863 disk.dm-1.out‚ 154.12890625 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚94.20113563537598 disk.rbd1.out‚ 2.146484375 disk.sda.out‚ 154.12890625# proc.memory.rssMB‚ 2873.83203125 disk.rbd3.out‚0 disk.rbd1.in‚0 network.sent‚ 1.1917111e+07% disk./.usageGB‚1110.2875785827637 disk.nvme0n1.in‚ 162.94140625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 94.1748046875 network.recv‚ 1.530159e+06 disk.sda1.in‚0 disk.sda3.out‚ 154.12890625 disk.rbd2.in‚0 disk.sda.in‚ 0.01171875' proc.memory.availableMB‚ 950.0859375 proc.cpu.threads‚17bUÍO©‚(:’ Á¹æÔŽ…ÒÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚163.27' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.653999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öômFª‚(:» Á¹æÔ‘ÖÁØ network.recv‚ 1.536577e+06# disk./.usageGB‚1110.28763961792 disk.rbd1.out‚ 2.146484375 disk.rbd3.in‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚16977.6181640625 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 154.6015625 network.sent‚ 1.2025077e+07 disk.rbd3.out‚0% disk.nvme0n1.out‚16977.6181640625 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625 disk.sda2.in‚0( proc.memory.availableMB‚ 950.99609375# proc.memory.rssMB‚ 2873.83203125 disk.sda1.out‚0) disk./.usagePercent‚74.51233508813868 disk.rbd0.out‚0 disk.rbd2.out‚ 94.3154296875 disk.drbd1777.out‚0 disk.dm-1.out‚ 154.6015625 cpu‚25.360763481026417 disk.sda.out‚ 154.6015625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 162.94140625$ memory_percent‚94.19558048248291* proc.memory.percent‚17.540478706359863 disk.rbd1.in‚0" disk.dm-0.out‚16977.6181640625 proc.cpu.threads‚17…¼T®2«- _wandbruntime‚4137  _runtime‚4137^^\vɬà ' lr-AdamW/pg1‚0.00012716677041687746 ' lr-AdamW/pg2‚0.00012716677041687746 trainer/global_step‚5299 % _timestamp‚1.7884520463412144e+09 _runtime‚4137.895635645 _step‚432iÃ:Î­È ' lr-AdamW/pg1‚0.00012716677041687746 ' lr-AdamW/pg2‚0.00012716677041687746 trainer/global_step‚5299 % _timestamp‚1.7884520463412144e+09 _runtime‚4137.895635645 _step‚432°%ÿq2®- _wandbruntime‚4137  _runtime‚4137ó,-ɯÛO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70284225 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77015510 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436980744192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77015508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100023816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436980744192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5299 _runtime‚4137.896611331 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77015510 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436980744192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77015508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100023816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436980744192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5299 _runtime‚4137.896611331±18í¾2±- _wandbruntime‚4138  _runtime‚4138…y«Oê&²äM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308025520673792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299956523917312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312462664483328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884520470331452e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100042296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77029526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4420808320888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚434 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2* eý ³éM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304377986676164 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4420808320888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚434 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308025520673792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336375876571136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299956523917312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312462664483328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308025520673792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308025538676736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884520470331452e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100042296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77029526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312463338862080 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FߤÇNº‚(:à ߹æÔ—ÒÙ disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1110.2876434326172 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 155.84765625 disk.rbd1.out‚ 2.146484375# proc.memory.rssMB‚ 2873.83203125 network.recv‚ 1.546897e+06 disk.rbd2.in‚0 disk.rbd2.out‚94.42138671875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚17075.48486328125 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875* proc.memory.percent‚17.540478706359863 network.sent‚ 1.2139182e+07 disk.sda1.out‚0& disk.nvme0n1.out‚17075.48486328125 disk.dm-1.out‚ 155.84765625$ memory_percent‚94.19534206390381" disk.nvme0c0n1.in‚ 162.94140625) disk./.usagePercent‚74.51233534414622 disk.sda.out‚ 155.84765625 disk.sda1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 951.03515625 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 cpu‚25.35851926873763( disk.nvme0c0n1.out‚17075.48486328125 disk.sda2.out‚05Ñž»‚(:“ î¹æÔõ¸¿Õ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚161.561' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.312200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¸ˆN¼‚(:à î¹æÔÈÀèØ disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 disk.sda1.out‚0$ memory_percent‚94.20056343078613' proc.memory.availableMB‚ 950.1796875 network.sent‚ 1.2145328e+07 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0# disk.dm-0.out‚17075.77783203125 disk.dm-1.out‚ 156.31640625 cpu‚25.357367025923384 proc.cpu.threads‚17( disk.nvme0c0n1.out‚17075.77783203125 disk.sda.in‚ 0.01171875* proc.memory.percent‚17.540478706359863) disk./.usagePercent‚74.51225726184828 disk.nvme0n1.in‚ 162.94140625 disk.rbd1.out‚ 2.146484375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1110.2864799499512 disk.rbd0.out‚0 disk.rbd2.out‚94.42138671875& disk.nvme0n1.out‚17075.77783203125 disk.drbd1777.in‚0 disk.rbd1.in‚0 network.recv‚ 1.551232e+06 disk.sda3.out‚ 156.31640625 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 156.31640625 disk.sda1.in‚0# proc.memory.rssMB‚ 2873.83203125XŒñ2½- _wandbruntime‚4175  _runtime‚4175¿• Rɾà _runtime‚4175.340162189 _step‚436 ' lr-AdamW/pg1‚0.00012593283005786265 ' lr-AdamW/pg2‚0.00012593283005786265 trainer/global_step‚5349 % _timestamp‚1.7884520837857819e+09ÌjîÎ¿È _step‚436 ' lr-AdamW/pg1‚0.00012593283005786265 ' lr-AdamW/pg2‚0.00012593283005786265 trainer/global_step‚5349 % _timestamp‚1.7884520837857819e+09 _runtime‚4175.340162189´|µò2À- _wandbruntime‚4175  _runtime‚4175íWüà'ÁÚO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100929830 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4477310187520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚5349 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4461481663088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77716308 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788452083786567e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344093831328768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339396762716672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚4175.341180817 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚315264804130304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25577680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376µxÞ$Ž2Ã- _wandbruntime‚4176  _runtime‚4176VUÈ$ ÄåM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884520844826465e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310843172616192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0El¾ð&ÅêM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884520844826465e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4478129615360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310843172616192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284¶äGQ2Æ- _wandbruntime‚4176  _runtime‚4176E<ån[ÇÕ /loss/FRt1/assignment_loss‚3.315155911034551 /loss/FRt4/assignment_loss‚4.243937380852238 2loss/SRqqt1/assignment_loss‚1.7345438332242125 2loss/SRqqt3/assignment_loss‚1.4698362058284236 1loss/FBt1/assignment_loss‚0.40171918021911146 .loss/FBt1/detection_loss‚2.276622166236242 1loss/SRqqt2/detection_loss‚1.2695574843409387 /loss/FBt2/detection_loss‚0.6257779376088365 1loss/FBt2/assignment_loss‚0.18117749844781433 /loss/FRt1/detection_loss‚2.1506699323654175 /loss/FRt3/detection_loss‚2.1896170397599537 /loss/FRt4/detection_loss‚3.0231657605017386 trainer/global_step‚5349 .loss/FBt3/detection_loss‚2.533349834382534 .loss/FBt4/detection_loss‚3.031213939189911 _runtime‚4176.039574234 _step‚439 0loss/FBt3/assignment_loss‚0.9571318682283163 1loss/SRqqt1/detection_loss‚1.6420810504579078 1loss/SRqqt4/detection_loss‚2.9273752686770065 epoch‚4 $ _timestamp‚1.788452084484752e+09 1loss/FBt4/assignment_loss‚0.23096192533537874 /loss/FRt2/detection_loss‚1.5749050823391462 0loss/FRt2/assignment_loss‚1.8717727133485138 2loss/SRqqt2/assignment_loss‚0.9057117128095618 2loss/SRqqt4/assignment_loss‚1.6811225563287735 %loss/total_loss‚45.75675492308429 /loss/FRt3/assignment_loss‚3.616942530208164 1loss/SRqqt3/detection_loss‚1.9024061113595963¨Ö`ÈÚ /loss/FRt1/assignment_loss‚3.315155911034551 /loss/FRt4/assignment_loss‚4.243937380852238 2loss/SRqqt1/assignment_loss‚1.7345438332242125 2loss/SRqqt3/assignment_loss‚1.4698362058284236 1loss/FBt1/assignment_loss‚0.40171918021911146 .loss/FBt1/detection_loss‚2.276622166236242 1loss/SRqqt2/detection_loss‚1.2695574843409387 /loss/FBt2/detection_loss‚0.6257779376088365 1loss/FBt2/assignment_loss‚0.18117749844781433 /loss/FRt1/detection_loss‚2.1506699323654175 /loss/FRt3/detection_loss‚2.1896170397599537 /loss/FRt4/detection_loss‚3.0231657605017386 trainer/global_step‚5349 .loss/FBt3/detection_loss‚2.533349834382534 .loss/FBt4/detection_loss‚3.031213939189911 _runtime‚4176.039574234 _step‚439 0loss/FBt3/assignment_loss‚0.9571318682283163 1loss/SRqqt1/detection_loss‚1.6420810504579078 1loss/SRqqt4/detection_loss‚2.9273752686770065 epoch‚4 $ _timestamp‚1.788452084484752e+09 1loss/FBt4/assignment_loss‚0.23096192533537874 /loss/FRt2/detection_loss‚1.5749050823391462 0loss/FRt2/assignment_loss‚1.8717727133485138 2loss/SRqqt2/assignment_loss‚0.9057117128095618 2loss/SRqqt4/assignment_loss‚1.6811225563287735 %loss/total_loss‚45.75675492308429 /loss/FRt3/assignment_loss‚3.616942530208164 1loss/SRqqt3/detection_loss‚1.9024061113595963·ÄvŸžÉ‚(:“ ý¹æÔئõÕ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚150.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.182400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÞÐHÊ‚(:½ ý¹æÔŸý™Ø disk.dm-1.out‚ 156.79296875 disk.rbd1.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1110.2884559631348& proc.memory.availableMB‚ 950.140625 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0" disk.dm-0.out‚17173.7763671875 disk.rbd1.out‚ 2.146484375$ memory_percent‚94.20080184936523# proc.memory.rssMB‚ 2873.83203125 cpu‚25.356339041716364 disk.drbd1777.in‚0' disk./.usagePercent‚74.512389873751 disk.rbd2.in‚0 disk.rbd2.out‚96.07373046875 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda.out‚ 156.79296875 disk.sda1.out‚0% disk.nvme0n1.out‚17173.7763671875 network.sent‚ 1.2151029e+07" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.540478706359863 network.recv‚ 1.555729e+06' disk.nvme0c0n1.out‚17173.7763671875 disk.dm-0.in‚ 162.94140625 disk.sda3.out‚ 156.79296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17BÎX’Ë‚(:‡ ŒºæÔóÝÒÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚92.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌPšUIÌ‚(:¾ ŒºæÔÖβÙ" disk.dm-0.out‚17271.3251953125 disk.sda1.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚96.07373046875" disk.nvme0c0n1.in‚ 162.94140625 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.2259694e+07 disk.sda.out‚ 157.66015625 disk.sda1.in‚0 disk.drbd1777.out‚0# memory_percent‚94.1978931427002 disk.rbd1.out‚ 2.146484375 disk.rbd0.out‚0' proc.memory.availableMB‚ 950.6171875 network.recv‚ 1.562092e+06 disk.sda2.out‚0 disk.sda3.out‚ 157.66015625% disk.nvme0n1.out‚17271.3251953125 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚17271.3251953125# proc.memory.rssMB‚ 2873.83203125 cpu‚25.355236838656577 disk.nvme0n1.in‚ 162.94140625) disk./.usagePercent‚74.51238628964553 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1110.288402557373 disk.sda2.in‚0 disk.dm-1.out‚ 157.66015625* proc.memory.percent‚17.540478706359863—Ã(h2Í- _wandbruntime‚4212  _runtime‚4212"qýÉÎà _runtime‚4212.764282247 _step‚440 ' lr-AdamW/pg1‚0.00012469184777928445 ' lr-AdamW/pg2‚0.00012469184777928445 trainer/global_step‚5399 % _timestamp‚1.7884521212098649e+09GÍj§ÎÏÈ ' lr-AdamW/pg1‚0.00012469184777928445 ' lr-AdamW/pg2‚0.00012469184777928445 trainer/global_step‚5399 % _timestamp‚1.7884521212098649e+09 _runtime‚4212.764282247 _step‚440¸hëÛŽ2Ð- _wandbruntime‚4212  _runtime‚4212Œv1)$ÑÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78417110 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180293030 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4501662564740 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101875920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4518295993344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342479299599872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342479323539968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 _runtime‚4212.765248145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 trainer/global_step‚5399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313604471519744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101855882 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚239öLØ·40096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180272990 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4518295993344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788452121210648e+09÷Y'^å'ÒßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78417110 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180293030 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4501662564740 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101875920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4518295993344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342479299599872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342479323539968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4502317002496 _runtime‚4212.765248145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 trainer/global_step‚5399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313604471519744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101855882 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180272990 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4518295993344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788452121210648e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872¹Ì—"Ö2Ó- _wandbruntime‚4213  _runtime‚4213ïƒxŸë&ÔåM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313660824558592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4740967182848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309949398933656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚5399 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632ï1R‹ð&ÕêM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313660824558592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4740967182848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309949398933656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚5399 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519771796992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342540974277632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ºP‰ 2Ö- _wandbruntime‚4213  _runtime‚4213ºåŸbW×Ñ 2loss/SRqqt3/assignment_loss‚1.7401091848810515 0loss/FBt1/assignment_loss‚1.1180298270450697 /loss/FRt1/detection_loss‚2.1896561195655746 1loss/SRqqt4/detection_loss‚2.4440471192200977 /loss/FRt2/assignment_loss‚1.664875254034996 1loss/SRqqt4/assignment_loss‚2.358435540397962 /loss/FBt4/assignment_loss‚0.329375136323506 1loss/SRqqt3/detection_loss‚1.7246748939156533 _runtime‚ 4213.45865162 /loss/FBt1/detection_loss‚2.0690208077430725 trainer/global_step‚5399 2loss/SRqqt1/assignment_loss‚1.5900833351936723 0loss/FBt3/assignment_loss‚0.0905132376695595 .loss/FRt4/detection_loss‚3.429301852529699 1loss/SRqqt1/detection_loss‚1.8490164772208249 1loss/SRqqt2/detection_loss‚1.4971635204824534 0loss/FRt1/assignment_loss‚3.0581327847072055 1loss/FBt2/assignment_loss‚0.18213782308808524 /loss/FRt2/detection_loss‚1.9134066401359935 .loss/FBt3/detection_loss‚3.418915713826815 .loss/FRt4/assignment_loss‚3.52705055475235 2loss/SRqqt2/assignment_loss‚1.1922396208413624 .loss/FRt3/detection_loss‚2.206687055072006 %loss/total_loss‚46.21758804501565 % _timestamp‚1.7884521219043405e+09 _step‚443 0loss/FRt3/assignment_loss‚3.1386762066763274 /loss/FBt2/detection_loss‚0.6397516793915852 /loss/FBt4/detection_loss‚2.8462876603007317 epoch‚4T/–ØÖ 2loss/SRqqt2/assignment_loss‚1.1922396208413624 .loss/FRt3/detection_loss‚2.206687055072006 %loss/total_loss‚46.21758804501565 % _×l ûÆtimestamp‚1.7884521219043405e+09 _step‚443 0loss/FRt3/assignment_loss‚3.1386762066763274 /loss/FBt2/detection_loss‚0.6397516793915852 /loss/FBt4/detection_loss‚2.8462876603007317 epoch‚4 2loss/SRqqt3/assignment_loss‚1.7401091848810515 0loss/FBt1/assignment_loss‚1.1180298270450697 /loss/FRt1/detection_loss‚2.1896561195655746 1loss/SRqqt4/detection_loss‚2.4440471192200977 /loss/FRt2/assignment_loss‚1.664875254034996 1loss/SRqqt4/assignment_loss‚2.358435540397962 /loss/FBt4/assignment_loss‚0.329375136323506 1loss/SRqqt3/detection_loss‚1.7246748939156533 _runtime‚ 4213.45865162 /loss/FBt1/detection_loss‚2.0690208077430725 trainer/global_step‚5399 2loss/SRqqt1/assignment_loss‚1.5900833351936723 0loss/FBt3/assignment_loss‚0.0905132376695595 .loss/FRt4/detection_loss‚3.429301852529699 1loss/SRqqt1/detection_loss‚1.8490164772208249 1loss/SRqqt2/detection_loss‚1.4971635204824534 0loss/FRt1/assignment_loss‚3.0581327847072055 1loss/FBt2/assignment_loss‚0.18213782308808524 /loss/FRt2/detection_loss‚1.9134066401359935 .loss/FBt3/detection_loss‚3.418915713826815 .loss/FRt4/assignment_loss‚3.52705055475235»å¦×›’Ù‚(:‡ ›ºæÔÔËÍÕ gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e[ÓÜBÚ‚(:· ›ºæÔ¿ÄØ network.recv‚ 1.565979e+06 disk.rbd1.out‚ 2.146484375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0" disk.dm-0.out‚17271.8349609375) disk./.usagePercent‚74.51238808169826 disk.dm-1.out‚ 158.203125 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2873.83203125 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 158.203125" disk.nvme0c0n1.in‚ 162.94140625* proc.memory.percent‚17.540478706359863 cpu‚25.354167757809787 proc.cpu.threads‚17 network.sent‚ 1.226497e+07 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd2.out‚96.21435546875 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 162.94140625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚17271.8349609375 disk.drbd1777.out‚0& proc.memory.availableMB‚ 950.359375$ disk./.usageGB‚1110.288429260254 disk.rbd3.in‚0 disk.sda.out‚ 158.203125 disk.rbd2.in‚0% disk.nvme0n1.out‚17271.8349609375$ memory_percent‚94.199466705322272ÛïžÛ‚(:“ ªºæÔþ—ùÕ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.401' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.080199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wIµNÜ‚(:à ªºæÔ¤¢ÊØ proc.cpu.threads‚17 disk.dm-0.in‚ 162.94140625 disk.rbd1.out‚ 2.146484375 disk.rbd3.in‚0 disk.sda.out‚ 158.61328125" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚17373.56494140625& disk.nvme0n1.out‚17373.56494140625 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 158.61328125 disk.rbd2.in‚0$ memory_percent‚94.20125484466553% disk./.usageGB‚1110.2922325134277 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 950.06640625 cpu‚25.35315388554448 network.sent‚ 1.2373703e+07) disk./.usagePercent‚74.51264332120991 disk.rbd3.out‚0* proc.memory.percent‚17.540478706359863 network.recv‚ 1.572251e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd2.out‚96.21435546875# disk.dm-0.out‚17373.56494140625 disk.sda2.in‚0 disk.dm-1.out‚ 158.61328125# proc.memory.rssMB‚ 2873.83203125Õksi’Ý‚(:‡ ¹ºæÔ•ÈÎÕ gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=°l[IÞ‚(:¾ ¹ºæÔ¾«¯Ø( proc.memory.availableMB‚ 950.30859375* proc.memory.percent‚17.540478706359863% disk./.usageGB‚1110.2922973632812 disk.dm-0.in‚ 162.94140625 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 96.31640625 cpu‚25.352091699537034' disk./.usagePercent‚74.512647673338 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 159.16796875" disk.nvme0c0n1.in‚ 162.94140625& disk.nvme0n1.out‚17471.83251953125 disk.sda2.in‚0 disk.dm-1.out‚ 159.16796875 disk.rbd3.in‚0 disk.sda.out‚ 159.16796875 disk.sda1.in‚0 disk.drbd1777.out‚0# memory_percent‚94.1997766494751# proc.memory.rssMB‚ 2873.83203125 network.sent‚ 1.2379294e+07 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17471.83251953125 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 network.recv‚ 1.576448e+06# disk.dm-0.out‚17471.83251953125 disk.rbd1.out‚ 2.146484375’‘g2ß- _wandbruntime‚4250  _runtime‚4250ìã_…Èà _runtime‚4250.123864491 _step‚444 ' lr-AdamW/pg1‚0.00012344408219998971 ' lr-AdamW/pg2‚0.00012344408219998971 trainer/global_step‚5449 $ _timestamp‚1.788452158569352e+09~ä8ªÍáÇ ' lr-AdamW/pg1‚0.00012344408219998971 ' lr-AdamW/pg2‚0.00012344408219998971 trainer/global_step‚5449 $ _timestamp‚1.788452158569352e+09 _runtime‚4250.123864491 _step‚444¼Öþk2â- _wandbruntime‚4250  _runtime‚4250p;¾ºà'ãÚO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102801972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316422141465088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316422141465088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102801972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚445 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102781934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72195814 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316422123462144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181899842 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26035958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908[/ïå'äßO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181919882 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350345393842688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316422123462144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181899842 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320982079626752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26035958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102801972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316422141465088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316422141465088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4542497904148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102801972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚445 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102781934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4559938161664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72195814 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345561836483072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46168478 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884521592756286e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181932312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316478476500992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.o¨S¤!¥n_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚5449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350407699114496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚446 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0—å¼$ð&çêM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4560757602816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚5449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350407699114496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚446 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46168478 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884521592756286e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181932312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316478476500992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583987681312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4826573886976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884521960844193e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323840717374976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319239793407488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856°’Ÿ~ôàO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319239775404544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103707986 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319239775404544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_tÈ|XÙhrain_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583987681312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4826573886976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884521960844193e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323840717374976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319239793407488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315465762102948 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4287.639108912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310882445898752 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5499 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601087032832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46576878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788452196811129e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319296146446336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4601743408640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348706024103936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103726440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319296128443392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4288.366146717 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26269610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827430836224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353533478880256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315522154145796 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323897215476224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72846488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26269161 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79832724 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310937350029312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315521482668472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5499 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601087032832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46576878 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788452196811129e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827269322752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319296146446336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310937332989952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4601743408640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4601743408640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348706024103936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103726440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319296128443392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4288.366146717 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26269610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827430836224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353533478880256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315522154145796 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ÂL@2ø- _wandbruntime‚4288  _runtime‚4288,Ž VùÐ 0loss/FRt2/assignment_loss‚1.5595697959264119 1loss/FBt2/assignment_loss‚0.19219058127236419 /loss/FRt2/detection_loss‚1.8842551369220017 %loss/total_loss‚45.05653000604306 0loss/FRt3/assignment_loss‚2.8439989306709985 /loss/FBt2/detection_loss‚0.6601525782200987 /loss/FBt4/detection_loss‚2.5513496073809536 epoch‚4 0loss/FBt1/assignment_loss‚0.4504636848966281 /loss/FRt4/assignment_loss‚4.275206102265252 2loss/SRqqt4/assignment_loss‚1.8281862353285154 0loss/FBt4/assignment_loss‚0.5626911486245014 0loss/SRqqt3/detection_loss‚2.063988966246446 1loss/SRqqt2/assignment_loss‚1.341199861041137 ,loss/FRt3/detection_loss‚2.1900767555291 .loss/FRt1/detection_loss‚2.314084836244583 /loss/FRt4/detection_loss‚3.0192874868710837 1loss/SRqqt2/detection_loss‚0.8746936025896243 1loss/SRqqt4/detection_loss‚2.6017994344234467 trainer/global_step‚5499 0loss/FRt1/assignment_loss‚3.2100202798843385 0loss/SRqqt1/detection_loss‚1.865489334452386 .loss/FBt1/detection_loss‚2.758397251367569 .loss/FBt3/detection_loss‚2.659788653254509 _runtime‚4288.367319334 _step‚451 2loss/SRqqt1/assignment_loss‚1.6638194154491466 2loss/SRqqt3/assignment_loss‚1.3032465342964445 0loss/FBt3/assignment_loss‚0.3825737928855233 % _timestamp‚1.7884521968130176e+09«shÈ[úÕ _runtime‚4288.367319334 _step‚451 2loss/SRqqt1/assignment_loss‚1.6638194154491466 2loss/SRqqt3/assignment_loss‚1.3032465342964445 0loss/FBt3/assignment_loss‚0.3825737928855233 % _timestamp‚1.7884521968130176e+09 0loss/FRt2/assignment_loss‚1.5595697959264119 1loss/FBt2/assignment_loss‚0.19219058127236419 /loss/FRt2/detection_loss‚1.8842551369220017 %loss/total_loss‚45.05653000604306 0loss/FRt3/assignment_loss‚2.8439989306709985 /loss/FBt2/detection_loss‚0.6601525782200987 /loss/FBt4/detection_loss‚2.5513496073809536 epoch‚4 0loss/FBt1/assignment_loss‚0.4504636848966281 /loss/FRt4/assignment_loss‚4.275206102265252 2loss/SRqqt4/assignment_loss‚1.8281862353285154 0loss/FBt4/assignment_loss‚0.5626911486245014 0loss/SRqqt3/detection_loss‚2.063988966246446 1loss/SRqqt2/assignment_loss‚1.341199861041137 ,loss/FRt3/detection_loss‚2.1900767555291 .loss/FRt1/detection_loss‚2.314084836244583 /loss/FRt4/detection_loss‚3.0192874868710837 1loss/SRqqt2/detection_loss‚0.8746936025896243 1loss/SRqqt4/detection_loss‚2.6017994344234467 trainer/global_step‚5499 0loss/FRt1/assignment_loss‚3.2100202798843385 0loss/SRqqt1/detection_loss‚1.865489334452386 .loss/FBt1/detection_loss‚2.758397251367569 .loss/FBt3/detection_loss‚2.659788653254509Ã9Èõ’û‚(:‡ æºæÔìöÅ× gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛùdWFü‚(:» æºæÔ¼‡‰Ù disk.rbd2.out‚97.96875" disk.nvme0c0n1.in‚ 162.94140625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚94.19996738433838# proc.memory.rssMB‚ 2873.83203125* proc.memory.percent‚17.540478706359863 disk.rbd2.in‚0% disk.nvme0n1.out‚17569.9404296875" disk.dm-0.out‚17569.9404296875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875% disk./.usageGB‚1110.2922706604004 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.2499562e+07 disk.dm-0.in‚ 162.94140625 disk.sda3.out‚ 160.93359375( proc.memory.availableMB‚ 950.27734375 network.recv‚ 1.591244e+06) disk./.usagePercent‚74.51264588128525 disk.nvme0n1.in‚ 162.94140625 disk.sda.out‚ 160.93359375' disk.nvme0c0n1.out‚17569.9404296875 disk.drbd1777.out‚0 disk.dm-1.out‚ 160.93359375 disk.rbd3.in‚0 cpu‚25.348991040399017 disk.rbd0.out‚0A Œý‚(:‡ õºæÔàÙÕ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErroÒØëèrs‚0OTÓBþ‚(:· õºæÔýûüÙ) disk./.usagePercent‚74.51264613729278 disk.rbd1.out‚ 2.146484375 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625% disk.nvme0c0n1.out‚17667.69140625 cpu‚25.348002215839916 disk.sda1.in‚0 disk.dm-0.out‚17667.69140625$ memory_percent‚94.19927597045898& proc.memory.availableMB‚ 950.390625* proc.memory.percent‚17.540478706359863 disk.dm-1.out‚ 161.60546875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.dm-0.in‚ 162.94140625 network.recv‚ 1.5973e+06 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 161.60546875 disk.rbd0.in‚0% disk./.usageGB‚1110.2922744750977# disk.nvme0n1.out‚17667.69140625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 161.60546875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2873.83203125 network.sent‚ 1.2607659e+07 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚98.12158203125 disk.nvme0n1.in‚ 162.94140625Sï™ÿ‚(:’ „»æÔªúßÕ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.889999999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K¸ÌE€‚(:º „»æÔ§“øØ) disk./.usagePercent‚74.51264741733046 disk.rbd2.out‚98.13720703125# disk.nvme0n1.out‚17765.25390625 disk.dm-0.out‚17765.25390625 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 network.sent‚ 1.2613938e+07 disk.dm-0.in‚ 162.94140625 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2873.83203125* proc.memory.percent‚17.540478706359863 network.recv‚ 1.601945e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 162.32421875( proc.memory.availableMB‚ 949.89453125 cpu‚25.346928662818446 disk.nvme0n1.in‚ 162.94140625 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚94.20230388641357 disk.sda3.out‚ 162.32421875% disk.nvme0c0n1.out‚17765.25390625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 162.32421875 disk.rbd1.out‚ 2.146484375$ disk./.usageGB‚1110.292293548584 disk.sda2.out‚0 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625Úhx€2- _wandbruntime‚4324  _runtime‚4324 âJÅ‚¿ % lr-AdamW/pg1‚0.000120929242628596 % lr-AdamW/pg2‚0.000120929242628596 trainer/global_step‚5549 % _timestamp‚1.7884522333028576e+09 _runtime‚4324.857249782 _step‚452GÊ"6ÊƒÄ % _timestamp‚1.7884522333028576e+09 _runtime‚4324.857249782 _step‚452 % lr-AdamW/pg1‚0.000120929242628596 % lr-AdamW/pg2‚0.000120929242628596 trainer/global_step‚5549ÄFñÓ_2„- _wandbruntime‚4324  _runtime‚4324'(pà'…ÚO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318252475447472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104654076 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104634038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46977108 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356596403943424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322057427346944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641253410816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80519510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26493979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641253410816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0‰;å'†ßO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26493979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641253410816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318252475447472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104654076 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104634038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46977108 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356596403943424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322057427346944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641253410816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104654076 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80519510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928Åâ*2‡- _wandbruntime‚4325  _runtime‚4325j'Ž~˜#ˆåM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326755853224448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80533526 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351788584927232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313682539517952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104672556 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351788560987136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4625639455892 I?DeviceStatsMonitor.on_train_batch_end/activemê ]S.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642072838656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104652492 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚4325.570917473îhTð&‰êM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351788584927232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313682539517952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104672556 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351788560987136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322113798388736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4642729214464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4625639455892 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642072838656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104652492 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚4325.570917473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚326755853224448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80533526 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884522717597277e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0׸ ÅÓ–àO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329557992871424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4682239216640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324875079289344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81220308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316372858954752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316372841915392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884522717597277e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912836025344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74108877 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47385510 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186780398 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81220310 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4682895579136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359722119534080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _runtime‚4363.314496669 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4682895579136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚457 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889Éð›`û2—- _wandbruntime‚4364  _runtime‚4364 56ë&˜åM P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47393678 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884522724845552e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4666474795300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 _step‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354871121810432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316427763085312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81234326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4683058644480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4683058644480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚5599 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47393678 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884522724845552e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4666474795300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4665820357336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 _step‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354871121810432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316427763085312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81234326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4683058644480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4683058644480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Êro,…2š- _wandbruntime‚4364  _runtime‚4364¹6ÀW›Ñ 2loss/SRqqt1/assignment_loss‚1.4401390787959099 0loss/FBt3/assignment_loss‚0.2469376031166086 epoch‚4 0loss/FRt2/assignment_loss‚1.0831615787286024 /loss/FRt4/assignment_loss‚4.140143486169668 0loss/FBt2/assignment_loss‚0.1845598952172599 0loss/SRqqt1/detection_loss‚1.709664363861084 %loss/total_loss‚43.84848714135397 _runtime‚ 4364.04135424 2loss/SRqqt4/assignment_loss‚1.8500615908549383 /loss/FBt4/assignment_loss‚0.954462960577355 /loss/FRt2/detection_loss‚1.7020097628522377 0loss/SRqqt4/detection_loss‚2.494765592309145 /loss/FBt2/detection_loss‚0.5293490722076968 .loss/FBt3/detection_loss‚2.203403977247385 trainer/global_step‚5599 1loss/FBt1/assignment_loss‚0.41593040004372595 1loss/SRqqt2/detection_loss‚1.0669211971689947 .loss/FBt4/detection_loss‚2.173107229746305 0loss/FRt1/assignment_loss‚2.4968935156410392 2loss/SRqqt3/assignment_loss‚1.5710065560681479 /loss/FRt1/detection_loss‚2.3706701628186484 /loss/FRt3/detection_loss‚2.0850466465463455 1loss/SRqqt3/detection_loss‚2.4068183847836084 /loss/FBt1/detection_loss‚3.1678093671798706 _step‚459 /loss/FRt3/assignment_loss‚3.521143845149449 % _timestamp‚1.7884522724869492e+09 2loss/SRqqt2/assignment_loss‚1.0513148215250112 .loss/FRt4/detection_loss‚2.983166052744939Òßj\œÖ 0loss/FRt2/assignment_loss‚1.0831615787286024 /loss/FRt4/assignment_loss‚4.140143486169668 0loss/FBt2/assignment_loss‚0.1845598952172599 0loss/SRqqt1/detection_loss‚1.709664363861084 %loss/total_loss‚43.84848714135397 _runtime‚ 4364.04135424 2loss/SRqqt4/assignment_loss‚1.8500615908549383 /loss/FBt4/assignment_loss‚0.954462960577355 /loss/FRt2/detection_loss‚1.7020097628522377 0loss/SRqqt4/detection_loss‚2.494765592309145 /loss/FBt2/detection_loss‚0.5293490722076968 .loss/FBt3/detection_loss‚2.203403977247385 trainer/global_step‚5599 1loss/FBt1/assignment_loss‚0.41593040004372595 1loss/SRqqt2/detection_loss‚1.0669211971689947 .loss/FBt4/detection_loss‚2.173107229746305 0loss/FRt1/assignment_loss‚2.4968935156410392 2loss/SRqqt3/assignment_loss‚1.5710065560681479 /loss/FRt1/detection_loss‚2.3706701628186484 /loss/FRt3/detection_loss‚2.0850466465463455 1loss/SRqqt3/detection_loss‚2.4068183847836084 /loss/FBt1/detection_loss‚3.1678093671798706 _step‚459 /loss/FRt3/assignment_loss‚3.521143845149449 % _timestamp‚1.7884522724869492e+09 2loss/SRqqt2/assignment_loss‚1.0513148215250112 .loss/FRt4/detection_loss‚2.983166052744939 2loss/SRqqt1/assignment_loss‚1.4401390787959099 0loss/FBt3/assignment_loss‚0.2469376031166086 epoch‚4Ëþ9Ö“‚(:ˆ ±»æÔ¤òë× gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þâVˆHž‚(:½ ±»æÔü¤ÀÚ disk.nvme0n1.in‚ 162.94140625 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2873.8359375 proc.cpu.threads‚17) disk./.usagePercent‚74.51265458554141 disk.rbd2.in‚0$ memory_percent‚94.19970512390137% disk./.usageGB‚1110.2924003601074 disk.rbd0.in‚0 disk.sda.out‚ 164.1484375' proc.memory.availableMB‚ 950.3203125* proc.memory.percent‚17.540502548217773 network.sent‚ 1.273242e+07 disk.rbd2.out‚99.91455078125 disk.dm-1.in‚ 0.01171875 network.recv‚ 1.61484e+06 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚17864.38427734375 disk.drbd1777.in‚0 disk.dm-1.out‚ 164.1484375 disk.sda1.out‚0 cpu‚25.344037417680497 disk.sda3.out‚ 164.1484375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875& disk.nvme0n1.out‚17864.38427734375 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚17864.38427734375 disk.sda2.in‚0 disk.rbd1.out‚ 2.146484375ÐÜO™“Ÿ‚(:ˆ À»æÔìÏÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¸I'E ‚(:º À»æÔïæÿØ network.sent‚ 1.28412e+07 disk.rbd2.in‚0( disk.nvme0c0n1.out‚17962.00634765625 disk.drbd1777.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚17962.00634765625( proc.memory.availableMB‚ 950.28515625* proc.memory.percent‚17.540526390075684( disk./.usagePercent‚74.5092353489141% disk./.usageGB‚1110.2414512634277 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.146484375 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2873.83984375 cpu‚25.34305436704977 network.recv‚ 1.621385e+06 disk.sda1.out‚0 disk.sda3.out‚ 164.859375 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.out‚99.93017578125" disk.nvme0c0n1.in‚ 162.94140625 disk.sda2.in‚0 disk.dm-1.out‚ 164.859375$ memory_percent‚94.19991970062256 disk.rbd1.in‚0 disk.sda.out‚ 164.859375 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚17962.00634765625 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 162.94140625 proc.cpu.threads‚17ãiž¡‚(:“ Ï»æÔòݸÕ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.041' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.008199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•càA¢‚(:¶ Ï»æÔÄÝüØ network.sent‚ 1.2846419e+07 disk.sda3.in‚ 0.01171875* proc.memory.percent‚17.540526390075684 disk.sda3.out‚ 165.62890625 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 162.94140625# proc.memory.rssMB‚ 2873.83984375 disk.drbd1777.out‚0) disk./.usagePercent‚74.50923944503465 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 network.recv‚ 1.625033e+06 disk.rbd2.out‚100.0830078125$ disk.nvme0c0n1.out‚ 17962.8046875$ memory_percent‚94.20011043548584 cpu‚25.34206031005914 proc.cpu.threads‚17$ disk./.usageGB‚1110.241512298584 disk.rbd3.out‚0 disk.dm-0.out‚ 17962.8046875" disk.nvme0n1.out‚ 17962.8046875 disk.dm-0.in‚ 162.94140625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 165.62890625 disk.rbd3.in‚0 disk.sda.out‚ 165.62890625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 162.94140625( proc.memory.availableMB‚ 950.25390625¬á3B2£- _wandbruntime‚4401  _runtime‚4401w–ÿÈȤ _step‚460 ' lr-AdamW/pg1‚0.00011839040758965157 ' lr-AdamW/pg2‚0.00011839040758965157 trainer/global_step‚5649 % _timestamp‚1.7884523097643943e+09 _runtime‚ 4401.31880959Ëóuñ괂 ' lr-AdamW/pg2‚0.00011839040758965157 trainer/global_step‚5649 % _timestamp‚1.7884523097643943e+09 _runtime‚ 4401.31880959 _step‚460 ' lr-AdamW/pg1‚0.00011839040758965157ÌSa4æ+¦- _wandbruntime‚4401  _runtime^KŽ‚4401O4¦á'§ÛO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4705839261780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4723881384960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106486142 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357891984015872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362847811031552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81921110 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884523097652476e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955827015680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332415956254208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚461 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26952054 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327692749234688è´ÌÕæ'¨àO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4723881384960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106486142 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357891984015872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362847811031552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81921110 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884523097652476e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955827015680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332415956254208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106506180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚461 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332416630619648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26952054 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327692749234688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4705839261780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ͥȵ2©- _wandbruntime‚4402  _runtime‚4402á¥" ê&ªäM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74758994 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4724700826112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4724044450304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323879608270696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81935126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106504596 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357953658693632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956916 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4707310134708 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332473803099648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752æQì«éM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4724700826112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4724044450304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323879608270696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81935126 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106504596 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502ª¨ê542848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357953658693632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956916 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4707310134708 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332473803099648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74758994 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362910304506368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323880279748020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414Î9QÏ2¬- _wandbruntime‚4402  _runtime‚4402åÿ½\­Ö % _timestamp‚1.7884523104626977e+09 2loss/SRqqt4/assignment_loss‚1.9993802149380957 /loss/FRt2/detection_loss‚1.9069236660997073 0loss/SRqqt4/detection_loss‚2.481386606182371 trainer/global_step‚5649 /loss/FRt4/assignment_loss‚3.658708914120992 1loss/FBt2/assignment_loss‚0.14954866865518285 .loss/FRt4/detection_loss‚2.928128095467885 0loss/FRt3/assignment_loss‚3.6380232771237693 1loss/FBt1/assignment_loss‚0.05052668802845043 .loss/FRt3/detection_loss‚2.405786630842421 1loss/SRqqt1/detection_loss‚1.6141330157978493 %loss/total_loss‚47.33366263258013 .loss/FBt4/detection_loss‚4.635492071509361 /loss/FRt1/assignment_loss‚2.875589745385306 1loss/SRqqt3/assignment_loss‚1.510246118492618 /loss/FBt1/detection_loss‚3.3087651133537292 /loss/FBt2/detection_loss‚0.7547454257111146 epoch‚4 _runtime‚4402.017037634 0loss/FRt2/assignment_loss‚1.9050951653056674 1loss/FBt3/assignment_loss‚0.19678317785015184 /loss/FRt1/detection_loss‚2.5724336771028384 1loss/SRqqt3/detection_loss‚2.0161163061857224 .loss/FBt3/detection_loss‚2.914030519398776 _step‚463 2loss/SRqqt1/assignment_loss‚1.5650306844659019 1loss/SRqqt2/assignment_loss‚1.001095032058972 3loss/FBt4/assignment_loss‚0.0033370992750789696 1loss/SRqqt2/detection_loss‚1.2423567192281706?Îêa®Û .loss/FBt4/detection_loss‚4.635492071509361 /loss/FRt1/assignment_loss‚2.875589745385306 1loss/SRqqt3/assignment_loss‚1.510246118492618 /loss/FBt1/detection_loss‚3.3087651133537292 /loss/FBt2/detection_loss‚0.7547454257111146 epoch‚4 _runtime‚4402.017037634 0loss/FRt2/assignment_loss‚1.9050951653056674 1loss/FBt3/assignment_loss‚0.19678317785015184 /loss/FRt1/detection_loss‚2.5724336771028384 1loss/SRqqt3/detection_loss‚2.0161163061857224 .loss/FBt3/detection_loss‚2.914030519398776 _step‚463 2loss/SRqqt1/assignment_loss‚1.5650306844659019 1loss/SRqqt2/assignment_loss‚1.001095032058972 3loss/FBt4/assignment_loss‚0.0033370992750789696 1loss/SRqqt2/detection_loss‚1.2423567192281706 % _timestamp‚1.7884523104626977e+09 2loss/SRqqt4/assignment_loss‚1.9993802149380957 /loss/FRt2/detection_loss‚1.9069236660997073 0loss/SRqqt4/detection_loss‚2.481386606182371 trainer/global_step‚5649 /loss/FRt4/assignment_loss‚3.658708914120992 1loss/FBt2/assignment_loss‚0.14954866865518285 .loss/FRt4/detection_loss‚2.928128095467885 0loss/FRt3/assignment_loss‚3.6380232771237693 1loss/FBt1/assignment_loss‚0.05052668802845043 .loss/FRt3/detection_loss‚2.405786630842421 1loss/SRqqt1/detection_loss‚1.6141330157978493 %loss/total_loss‚47.33366263258013Ï»5ë$“¯‚(:ˆ Þ»æÔûÉòÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ø=²H°‚(:½ Þ»æÔ²ê©Ø network.sent‚ 1.2852593e+07 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.drbd1777.in‚0 network.recv‚ 1.62913e+06 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 166.078125" disk.nvme0c0n1.in‚ 162.94140625( disk.nvme0c0n1.out‚18060.43994140625& disk.nvme0n1.out‚18060.43994140625 disk.drbd1777.out‚0 disk.dm-1.out‚ 166.078125) disk./.usagePercent‚74.50912705772711% disk./.usageGB‚1110.2398376464844 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 162.94140625 disk.dm-0.in‚ 162.94140625# disk.dm-0.out‚18060.43994140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 2.146484375$ memory_percent‚94.20011043548584( proc.memory.availableMB‚ 950.25390625* proc.memory.percent‚17.540526390075684 disk.sda.out‚ 166.078125 disk.rbd2.out‚100.0986328125 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2873.83984375 cpu‚25.341080165860806 proc.cpu.threads‚17 disk.rbd2.in‚0剒±‚(:‡ í»æÔÇ£ˆÖ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚94.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VU¨ÔK²‚(:À í»æÔÅáºØ disk.sda2.out‚0 network.sent‚ 1.2960666e+07 disk.dm-1.out‚ 166.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚ 162.94140625 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 166.77734375 disk.sda1.out‚0% disk./.usageGB‚1110.2399597167969 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 162.94140625 disk.rbd2.out‚101.7978515625% disk.nvme0n1.out‚18158.9013671875 disk.sda2.in‚0# memory_percent‚94.1967248916626( proc.memory.availableMB‚ 950.80859375* proc.memory.percent‚17.540526390075684 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 162.94140625" disk.dm-0.out‚18158.9013671875 cpu‚25.340153825728116 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875) disk./.usagePercent‚74.50913524996821 disk.rbd1.out‚ 2.146484375 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0' disk.nvme0c0n1.out‚18158.9013671875# proc.memory.rssMB‚ 2873.83984375 network.recv‚ 1.635049e+06 disk.rbd3.out‚0 disk.sda3.out‚ 166.77734375¡¹ß 2³- _wandbruntime‚4438  _runtime‚4438ç!È´Â ' lr-AdamW/pg2‚0.00011711265236398292 trainer/global_step‚5699 $ _timestamp‚1.788452347017758e+09 _runtime‚4438.572154152 _step‚464 ' lr-AdamW/pg1‚0.00011711265236398292E§7_굂 ' lr-AdamW/pg1‚0.00011711265236398292 ' lr-AdamW/pg2‚0.00011711265236398292 trainer/global_step‚5699 $ _timestamp‚1.788452347017758e+09 _runtime‚4438.572154152 _step‚464Ð= tÔ2¶- _wandbruntime‚4438  _runtime‚4438Ôy¸á'·ÛO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190054142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182550 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330510383174144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884523470188699e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚465 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4764210828288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4764210828288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360974520899072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48202308 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326610601049696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884523470188699e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚465 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107412194 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335275268367872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4764210828288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82621910 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4746674601188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4764210828288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330510401177088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360974520899072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48202308 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326610601049696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 49991ù…‰ÛÌ86647552 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190054142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430ч™}2¹- _wandbruntime‚4439  _runtime‚44399çyzë&ºåM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335331766469120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999869999104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82635924 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 trainer/global_step‚5699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82635926 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884523477336664e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4439.289019409 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ö Qtð&»êM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366036041635840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4748145474116 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321918176141312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27186702 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82635924 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5000044095488 trainer/global_step‚5699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4765030256128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82635926 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884523477336664e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4439.289019409 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335331766469120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4765030256128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999869999104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4787509940596 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_spl“ö%˜¼it.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4476.124959128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338133906116096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324608461499392 _step‚469 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5042091252224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 trainer/global_step‚5749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338133231750656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ãÓ÷ æ'ÊàO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4476.124959128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338133906116096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324608461499392 _step‚469 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5042091252224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83322710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 trainer/global_step‚5749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4805852996608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338133231750656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48610708 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83322710 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4787509940596 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333328053119488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605ÕAaPM2Ë- _wandbruntime‚4477  _runtime‚4477Ð?À ê&ÌäM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324663382669312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76036021 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324663365629952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5749 $ _timestamp‚1.788452385522909e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83336726 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333384388155392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108376764 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490ÂEÒÍéM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4806672437760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48618878 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48618876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108376764 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364118708519936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324663382669312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333384406158336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76036021 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324663365629952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all¹@³Ð.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83336724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4806016061952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5749 $ _timestamp‚1.788452385522909e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4806672437760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83336726 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333384388155392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108376764 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76660985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109284336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367139618605568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336145705061888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4846838802432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚5799 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4513.870186107 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2×Òæ'ÚàO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚5799 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4513.870186107 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27641875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336145687058944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76660530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5085431625728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76660985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109284336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367139618605568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336145705061888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4846838802432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430Ùû ÂU2Û- _wandbruntime‚4514  _runtime‚4514ÇìTÆ0ÜåM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884524230105193e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4514.565511418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚474 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109282752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336202040097792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847658243584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847001867776 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109302816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232Z6ïºð&ÝêM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341049716344320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884524230105193e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4514.565511418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚474 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109282752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336202040097792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847658243584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847001867776 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109302816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27646461 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Úg”iŠ2Þ- _wandbruntime‚4514  _runtime‚4514£™‘XßÒ 1loss/FBt2/assignment_loss‚0.14524948203284963 .loss/FRt3/detection_loss‚2.563357623560088 0loss/FRt1/assignment_loss‚3.2532464451260035 0loss/FRt2/assignment_loss‚1.5358065042579383 1loss/SRqqt2/assignment_loss‚1.091432238034904 /loss/FBt1/detection_loss‚2.2566746532917024 _runtime‚ 4514.56725356 2loss/SRqqt3/assignment_loss‚1.1719433233762782 1loss/SRqqt4/detection_loss‚2.5003438649639005 1loss/SRqqt3/detection_loss‚1.8384750944872696 /loss/FBt4/detection_loss‚3.4433907866477966 0loss/FBt1/assignment_loss‚0.4102902229351457 1loss/FBt4/assignment_loss‚0.27438254600080353 /loss/FRt2/detection_loss‚1.8294279891670795 epoch‚4 $ _timestamp‚1.788452423012929e+09 0loss/FRt3/assignment_loss‚2.9648308753967285 0loss/SRqqt4/assignment_loss‚2.17059041896174 /loss/FBt3/assignment_loss‚0.533349047973752 /loss/FRt1/detection_loss‚2.3268348806434207 1loss/SRqqt1/detection_loss‚1.7861146200786937 .loss/FBt3/detection_loss‚2.198743164539337 %loss/total_loss‚44.57706949206603 trainer/global_step‚5799 _step‚475 /loss/FRt4/assignment_loss‚4.050411825378736 2loss/SRqqt1/assignment_loss‚1.5241147393530066 /loss/FRt4/detection_loss‚3.0811282893021903 1loss/SRqqt2/detection_loss‚0.9877308440208435 /loss/FBt2/detection_loss‚0.6392000125358223 ´>ƒ]à× 1loss/SRqqt3/detection_loss‚1.8384750944872696 /loss/FBt4/detection_loss‚3.4433907866477966 0loss/FBt1/assignment_loss‚0.4102902229351457 1loss/FBt4/assignment_loss‚0.27438254600080353 /loss/FRt2/detection_loss‚1.8294279891670795 epoch‚4 $ _timestamp‚1.788452423012929e+09 0loss/FRt3/assignment_loss‚2.9648308753967285 0loss/SRqqt4/assignment_loss‚2.17059041896174 /loss/FBt3/assignment_loss‚0.533349047973752 /loss/FRt1/detection_loss‚2.3268348806434207 1loss/SRqqt1/detection_loss‚1.7861146200786937 .loss/FBt3/detection_loss‚2.198743164539337 %loss/total_loss‚44.57706949206603 trainer/global_step‚5799 _step‚475 /loss/FRt4/assignment_loss‚4.050411825378736 2loss/SRqqt1/assignment_loss‚1.5241147393530066 /loss/FRt4/detection_loss‚3.0811282893021903 1loss/SRqqt2/detection_loss‚0.9877308440208435 /loss/FBt2/detection_loss‚0.6392000125358223 1loss/FBt2/assignment_loss‚0.14524948203284963 .loss/FRt3/detection_loss‚2.563357623560088 0loss/FRt1/assignment_loss‚3.2532464451260035 0loss/FRt2/assignment_loss‚1.5358065042579383 1loss/SRqqt2/assignment_loss‚1.091432238034904 /loss/FBt1/detection_loss‚2.2566746532917024 _runtime‚ 4514.56725356 2loss/SRqqt3/assignment_loss‚1.1719433233762782 1loss/SRqqt4/detection_loss‚2.5003438649639005Û^ePžá‚(:“ ǼæÔíðÞÕ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.992' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.998399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á1©|Kâ‚(:À ǼæÔÖàøØ disk.dm-0.in‚ 167.5390625 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚94.20163631439209 proc.cpu.threads‚17 disk.sda.out‚ 170.91015625 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 170.91015625& disk.nvme0n1.out‚18459.46142578125( proc.memory.availableMB‚ 950.00390625 cpu‚25.334507202375608 disk.rbd2.in‚0( disk.nvme0c0n1.out‚18459.46142578125( disk./.usagePercent‚74.5092361169367 disk.drbd1777.in‚0# disk.dm-0.out‚18459.46142578125 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 167.5390625 network.sent‚ 1.3200738e+07 disk.dm-1.out‚ 170.91015625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 167.5390625# proc.memory.rssMB‚ 2873.83984375% disk./.usageGB‚1110.2414627075195 disk.rbd1.out‚ 2.146484375! disk.rbd2.out‚103.79443359375 network.recv‚ 1.66457e+06 disk.drbd1777.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.540526390075684 disk.sda2.out‚0 disk.rbd0.out‚0Ï1êæ“ã‚(:ˆ Ö¼æÔ㟮Õ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.6722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z+üFä‚(:» Ö¼æÔ÷ÝØ disk.rbd2.in‚0$ memory_percent‚94.20292377471924# proc.memory.rssMB‚ 2873.83984375) disk./.usagePercent‚74.50922971674835 disk.sda2.in‚0 cpu‚25.333627807156905 network.recv‚ 1.670823e+06 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0! disk.rbd2.out‚103.81005859375 network.sent‚ 1.3309327e+07 disk.dm-0.in‚ 167.5390625 disk.sda.out‚ 171.3671875 disk.sda1.out‚0" disk.dm-0.out‚18557.2451171875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0$ disk./.usageGB‚1110.241367340088 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 167.5390625 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.146484375' disk.nvme0c0n1.out‚18557.2451171875% disk.nvme0n1.out‚18557.2451171875( proc.memory.availableMB‚ 949.79296875* proc.memory.percent‚17.540526390075684 proc.cpu.threads‚17 disk.dm-1.out‚ 171.3671875 disk.rbd3.in‚0 disk.sda3.out‚ 171.3671875! disk.nvme0c0n1.in‚ 167.5390625ÔP#sžå‚(:“ å¼æÔ»ñäÕ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.911599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û˜¡‚Cæ‚(:¸ å¼æÔ…þÚØ disk.sda2.out‚0) disk./.usagePercent‚74.50910427305656 disk.rbd3.out‚0 disk.sda.out‚ 171.94921875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚18655.451171875 network.sent‚ 1.3314981e+07 disk.rbd1.out‚ 2.146484375 disk.drbd1777.out‚0 disk.sda3.in‚ 0.01171875 network.recv‚ 1.675239e+06 disk.rbd1.in‚0 disk.rbd2.out‚ 103.962890625 disk.nvme0n1.in‚ 167.5390625! disk.dm-0.out‚18655.451171875& proc.memory.availableMB‚ 949.796875 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 171.94921875 disk.dm-0.in‚ 167.5390625 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2873.83984375 cpu‚25.332642483519297 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 167.5390625& disk.nvme0c0n1.out‚18655.451171875 disk.dm-1.out‚ 171.94921875% disk./.usageGB‚1110.2394981384277 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚94.20289993286133* proc.memory.percent‚17.540526390075684 disk.drbd1777.in‚0ƒQû2ç- _wandbruntime‚4551  _runtime‚4551IoÇèÁ % _timestamp‚1.7884524598689232e+09 _runtime‚4551.423367033 _step‚476 & lr-AdamW/pg1‚0.0001132492343361705 & lr-AdamW/pg2‚0.0001132492343361705 trainer/global_step‚5849ƒ ÌéÆ _runtime‚4551.423367033 _step‚476 & lr-AdamW/pg1‚0.0001132492343361705 & lr-AdamW/pg2‚0.0001132492343361705 trainer/global_step‚5849 % _timestamp‚1.7884524598689232e+09Ü¥^…2ê- _wandbruntime‚4551  _runtime‚4551¡¼>dëÚO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330098891594752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27870305 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4869835057168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 _runtime‚4551.424218918 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370222131548672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 _step‚477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49427510 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84724308 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370222155488768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297813 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 _runtime‚4551.424218918 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370222131548672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128591019520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 _step‚477 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84724310 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4887168245760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49427510 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84724308 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338963339001344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77298258 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370222155488768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297813 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339019710043136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788452460563065e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110228868 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77311012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343908354092544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339019710043136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788452460563065e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110228868 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77311012 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343908354092544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4928154051584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337754097042212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4910670396576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788452496514161e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85425110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341781008946688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0· ôÍå'üßO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4928154051584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337754097042212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378476393800704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4910670396576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788452496514161e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85425110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341781008946688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341780990943744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373304668431872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28099414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378476583461888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337754768519328 _runtime‚4588.068806998 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 4588.74756462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28104374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeõ§õviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928973479424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341837343982592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚5899 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 4588.74756462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373366319169536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28104374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928973479424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341837343982592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85439124 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚5899 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349568457108992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344598660889088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344598642886144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚485 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344598660889088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112062492 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376387205315072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86125910 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ÒÕt/ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340540810386736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50244308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904oÀŒ®æ'ŽàO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340540810386736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50244308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78573286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28328978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381602003495424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349568457108992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344598660889088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚344598642886144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚485 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344598660889088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112062492 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376387205315072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86125910 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872å³à]­2- _wandbruntime‚4625  _runtime‚4625µ^*ë&åM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 _step‚486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86139924 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86139924 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4952322171156 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340595859475144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884525338943837e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344654995924992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376448856052736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335644208781312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4625.449407092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349625629588992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344654995924992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚5949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215641176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86139926 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928Ý”ô6ð&‘êM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 _step‚486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4951667733192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86139924 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86139926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28333975 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86139924 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4952322171156 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340595859475144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884525338943837e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344654995924992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376448856052736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335644208781312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4625.449407092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349625629588992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344654995924992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 trainer/global_step‚5949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5215641176576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86139926 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0æLÈ®K2’- _wandbruntime‚4625  _runtime‚4625V^–ôV“Ð 1loss/FBt3/assignment_loss‚0.14463382772387307 .loss/FRt1/detection_loss‚2.213544256053865 0loss/SRqqt4/detection_loss‚2.589403909185658 /loss/FBt2/detection_loss‚0.5475618073574031 _runtime‚ 4625.45074336 2loss/SRqqt1/assignment_loss‚1.2008362943306565 _step‚487 0loss/FRt1/assignment_loss‚3.0713014329473176 /loss/FRt2/assignment_loss‚1.666106902062893 .loss/FRt3/detection_loss‚1.857516076498563 /loss/FRt4/detection_loss‚3.0748905638853707 1loss/SRqqt1/detection_loss‚1.6105908710185601 0loss/SRqqt2/detection_loss‚1.114059440791607 -loss/FBt1/detection_loss‚2.30714727299554 trainer/global_step‚5949 2loss/SRqqt3/assignment_loss‚1.5665053459897376 0loss/SRqqt4/assignment_loss‚1.59066288302774 1loss/FBt1/assignment_loss‚0.15705829032231122 -loss/FRt2/detection_loss‚1.88627021163702 /loss/FBt3/detection_loss‚3.1831407129764555 &loss/total_loss‚41.958278343000316 epoch‚4 % _timestamp‚1.7884525338964388e+09 1loss/SRqqt3/detection_loss‚1.6526270842081623 .loss/FBt4/detection_loss‚2.496123969554901 /loss/FRt4/assignment_loss‚3.417919004956881 1loss/FBt4/assignment_loss‚0.19775712182693042 2loss/SRqqt2/assignment_loss‚0.9747668662773711 1loss/FBt2/assignment_loss‚0.09108001531677233 0loss/FRt3/assignment_loss‚3.3467741820547317 )<³”Õ /loss/FRt4/assignment_loss‚3.417919004956881 1loss/FBt4/assignment_loss‚0.19775712182693042 2loss/SRqqt2/assignment_loss‚0.9747668662773711 1loss/FBt2/assignment_loss‚0.09108001531677233 0loss/FRt3/assignment_loss‚3.3467741820547317 1loss/FBt3/assignment_lo‘¸}¶=ss‚0.14463382772387307 .loss/FRt1/detection_loss‚2.213544256053865 0loss/SRqqt4/detection_loss‚2.589403909185658 /loss/FBt2/detection_loss‚0.5475618073574031 _runtime‚ 4625.45074336 2loss/SRqqt1/assignment_loss‚1.2008362943306565 _step‚487 0loss/FRt1/assignment_loss‚3.0713014329473176 /loss/FRt2/assignment_loss‚1.666106902062893 .loss/FRt3/detection_loss‚1.857516076498563 /loss/FRt4/detection_loss‚3.0748905638853707 1loss/SRqqt1/detection_loss‚1.6105908710185601 0loss/SRqqt2/detection_loss‚1.114059440791607 -loss/FBt1/detection_loss‚2.30714727299554 trainer/global_step‚5949 2loss/SRqqt3/assignment_loss‚1.5665053459897376 0loss/SRqqt4/assignment_loss‚1.59066288302774 1loss/FBt1/assignment_loss‚0.15705829032231122 -loss/FRt2/detection_loss‚1.88627021163702 /loss/FBt3/detection_loss‚3.1831407129764555 &loss/total_loss‚41.958278343000316 epoch‚4 % _timestamp‚1.7884525338964388e+09 1loss/SRqqt3/detection_loss‚1.6526270842081623 .loss/FBt4/detection_loss‚2.496123969554901ç` ?y‘•‚(:† ¿½æÔÈ·»Õ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.36' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îk¯ïO–‚(:Ä ¿½æÔ‹¢ÙØ% disk./.usageGB‚1110.2439270019531 disk.dm-0.in‚ 167.5390625 network.recv‚ 1.704159e+06 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 167.5390625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 167.5390625& disk.nvme0n1.out‚18953.06787109375 disk.dm-1.out‚ 175.68359375( proc.memory.availableMB‚ 949.54296875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.39013671875 disk.sda.out‚ 175.68359375$ memory_percent‚94.20444965362549 disk.sda2.in‚0 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚18953.06787109375 disk.drbd1777.out‚0# disk.dm-0.out‚18953.06787109375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2873.83984375 cpu‚25.327238338450055) disk./.usagePercent‚74.50940149780382 disk.sda3.out‚ 175.68359375 disk.sda2.out‚0* proc.memory.percent‚17.540526390075684 network.sent‚ 1.3554926e+07! disk.rbd2.out‚105.79931640625 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17ƒêß´—‚(:’ νæÔðìÏÕ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.428800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e0>ËK˜‚(:À νæÔæ±µÙ% disk./.usageGB‚1110.2439079284668# disk.dm-0.out‚19050.60693359375 disk.sda.in‚ 0.01171875 disk.drbd1777.in‚0 network.recv‚ 1.71062e+06) disk./.usagePercent‚74.50940021776616 disk.sda.out‚ 176.4453125 disk.sda1.out‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚19050.60693359375 disk.nvme0n1.in‚ 167.5390625 disk.sda2.out‚0 disk.dm-1.out‚ 176.4453125 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0 network.sent‚ 1.3663554e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 2.39013671875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 176.4453125( proc.memory.availableMB‚ 949.29296875# proc.memory.rssMB‚ 2873.83984375! disk.rbd2.out‚105.81494140625! disk.nvme0c0n1.in‚ 167.5390625 cpu‚25.326394315459428 disk.rbd2.in‚0 disk.dm-0.in‚ 167.5390625 disk.rbd3.out‚0$ memory_percent‚94.20597553253174* proc.memory.percent‚17.540526390075684& disk.nvme0n1.out‚19050.60693359375Ù<ÅŸ2™- _wandbruntime‚4661  _runtime‚4661Ïɚà _step‚488 ' lr-AdamW/pg1‚0.00010934623371320299 ' lr-AdamW/pg2‚0.00010934623371320299 trainer/global_step‚5999 % _timestamp‚1.7884525694578612e+09 _runtime‚4661.012248317îá;Î›È trainer/global_step‚5999 % _timestamp‚1.7884525694578612e+09 _runtime‚4661.012248317 _step‚488 ' lr-AdamW/pg1‚0.00010934623371320299 ' lr-AdamW/pg2‚0.00010934623371320299èE¥n@2œ- _wandbruntime‚4661  _runtime‚4661ìúàà'ÚO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 $ _timestamp‚1.788452569458621e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5010782025728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992341075392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4991686637636 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5010782025728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚5999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 _step‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79210497 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4661.013274491 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ˆA¥å'žßO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4991686637636 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112968506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚352426420491776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5010782025728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚5999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86826710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 _step‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79210497 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚4661.013274491 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5257776248832 $ _timestamp‚1.788452569458621e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5010782025728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5010125663232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4992341075392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496é1Ò÷ 2Ÿ- _wandbruntime‚4661  _runtime‚4661´'I® äM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347472647867392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28561771 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011601466880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788452570138947e+09 _step‚490 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113007024 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347472665870336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86840724 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86840726 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199827684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384790223484416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112986960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceFëïÔ< StatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4661.694093843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50660878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993157510564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ù[ÖPï&¡éM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4993157510564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347472647867392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50660876 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28561771 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011601466880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788452570138947e+09 _step‚490 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5010945091072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28562197 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113007024 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347472665870336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5011601466880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5010945091072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86840724 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86840726 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199827684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384790223484416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112986960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347472647867392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4661.694093843 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50660878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352483592958464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343381901342552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352484267337216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066êuD 2¢- _wandbruntime‚4661  _runtime‚46612xöº^£Ø /loss/FBt4/detection_loss‚2.8448845062937056 trainer/global_step‚5999 /loss/FRt1/assignment_loss‚2.958841539753808 0loss/FRt4/assignment_loss‚3.8412581086158752 /loss/FBt2/detection_loss‚0.6493635104332126 2loss/SRqqt4/assignment_loss‚1.9882584452629088 &loss/total_loss‚43.414522119547655 % _timestamp‚1.7884525701414423e+09 /loss/FRt1/detection_loss‚2.3993835921088853 1loss/FBt4/assignment_loss‚0.35243190202683244 .loss/FRt4/detection_loss‚2.852856758981943 1loss/SRqqt2/detection_loss‚1.1631019366766089 0loss/SRqqt4/detection_loss‚2.274990233353206 _step‚491 2loss/SRqqt1/assignment_loss‚1.3521095957268368 2loss/SRqqt3/assignment_loss‚1.2619134720880538 1loss/FBt2/assignment_loss‚0.20138078303952825 /loss/FBt3/detection_loss‚3.3652904987335206 epoch‚4 _runtime‚4661.695803705 0loss/FRt2/assignment_loss‚1.5499475513185774 /loss/FRt3/assignment_loss‚2.991646970019621 2loss/SRqqt2/assignment_loss‚1.0574836895398017 0loss/FBt1/assignment_loss‚0.1609660619869818 1loss/FBt3/assignment_loss‚0.11555596524849535 1loss/SRqqt1/detection_loss‚1.6255889896642077 1loss/SRqqt3/detection_loss‚1.7568266708403826 /loss/FRt2/detection_loss‚1.8490804602125925 .loss/FRt3/detection_loss‚1.982295085402096 /loss/FBt1/detection_loss‚2.8190657922199795WÀO5c¤Ý 1loss/SRqqt2/detection_loss‚1.1631019366766089 0loss/SRqqt4/detection_loss‚2.274990233353206 _step‚491 2loss/SRqqt1/assignment_loss‚1.3521095957268368 2loss/SRqqt3/assignment_loss‚1.2619134720880538 1loss/FBt2/assignment_loss‚0.20138078303952825 /loss/FBt3/detection_loss‚3.3652904987335206 epoch‚4 _runtime‚4661.695803705 0loss/FRt2/assignment_loss‚1.5499475513185774 /loss/FRt3/assignment_loss‚2.991646970019621 2loss/SRqqt2/assignment_loss‚1.0574836895398017 0loss/FBt1/assignment_loss‚0.1609660619869818 1loss/FBt3/assignment_loss‚0.11555596524849535 1loss/SRqqt1/detection_loss‚1.6255889896642077 1loss/SRqqt3/detection_loss‚1.7568266708403826 /loss/FRt2/detection_loss‚1.8490804602125925 .loss/FRt3/detection_loss‚1.982295085402096 /loss/FBt1/detection_loss‚2.8190657922199795 /loss/FBt4/detection_loss‚2.8448845062937056 trainer/global_step‚5999 /loss/FRt1/assignment_loss‚2.958841539753808 0loss/FRt4/assignment_loss‚3.8412581086158752 /loss/FBt2/detection_loss‚0.6493635104332126 2loss/SRqqt4/assignment_loss‚1.9882584452629088 &loss/total_loss‚43.414522119547655 % _timestamp‚1.7884525701414423e+09 /loss/FRt1/detection_loss‚2.3993835921088853 1loss/FBt4/assignment_loss‚0.35243190202683244 .loss/FRt4/detection_loss‚2.852856758981943ë _’¥‚(:‡ ݽæÔÖæÕ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.689 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“bæ¼C¦‚(:¸ ݽæÔÿ²Ù! disk.rbd2.out‚105.87744140625 disk.dm-0.in‚ 167.5390625# proc.memory.rssMB‚ 2873.83984375) disk./.usagePercent‚74.50940636194697 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.sda.in‚ 0.01171875$ memory_percent‚94.20583248138428( proc.memory.availableMB‚ 949.31640625 cpu‚25.325460374446788 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 167.5390625$ disk.nvme0n1.out‚19051.505859375! disk.dm-0.out‚19051.505859375 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0% disk./.usageGB‚1110.2439994812012 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 177.01171875 disk.sda3.out‚ 177.01171875 proc.cpu.threads‚17 disk.sda.out‚ 177.01171875 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 167.5390625* proc.memory.percent‚17.540526390075684 network.sent‚ 1.366929e+07 network.recv‚ 1.714743e+06 disk.rbd1.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚19051.505859375,zùT’§‚(:‡ ì½æÔ‹çëÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öß‹ÐC¨‚(:¸ ì½æÔî×Ø! disk.rbd2.out‚107.40478515625 disk.nvme0n1.in‚ 167.5390625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 177.65234375( proc.memory.availableMB‚ 949.85546875 disk.sda3.out‚ 177.65234375 disk.sda.out‚ 177.65234375 cpu‚25.324539046366432 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd2.in‚0 network.sent‚ 1.3777099e+07 disk.rbd1.out‚2.453125 disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 167.5390625* proc.memory.percent‚17.540526390075684% disk./.usageGB‚1110.2439994812012 disk.sda3.in‚ 0.01171875$ disk.nvme0n1.out‚19148.994140625& disk.nvme0c0n1.out‚19148.994140625 disk.drbd1777.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚19148.994140625$ memory_percent‚94.20254230499268# proc.memory.rssMB‚ 2873.83984375 disk.dm-0.in‚ 167.5390625 disk.sda2.in‚0 network.recv‚ 1.72038e+06) disk./.usagePercent‚74.50940636194697 disk.rbd1.in‚0 disk.rbd3.out‚0ÖG‚’©‚(:‡ û½æÔá”ÉÕ gpu.0.gpu‚50.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬èrL@ª‚(:µ û½æÔ´ü¦Ù! disk.nvme0c0n1.in‚ 167.5390625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 178.40234375$ memory_percent‚94.20325756072998 network.sent‚ 1.3782353e+07! disk.dm-0.out‚19246.533203125 disk.sda2.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0& disk.nvme0c0n1.out‚19246.533203125 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 949.73828125 disk.nvme0n1.in‚ 167.5390625 disk.rbd1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.540550231933594% disk./.usageGB‚1110.2439422607422 disk.rbd0.out‚0$ disk.nvme0n1.out‚19246.533203125 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 178.40234375 proc.memory.rssMB‚ 2873.84375 network.recv‚ 1.723997e+06! disk.rbd2.out‚107.52978515625 disk.dm-1.out‚ 178.40234375 disk.rbd1.out‚2.453125 disk.sda.in‚ 0.01171875 cpu‚25.32466101129142 proc.cpu.threads‚17 disk.dm-0.in‚ 167.5390625) disk./.usagePercent‚74.50940252183396¡¶±m‘«‚(:† Š¾æÔß±íÕ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l ü¾<¬‚(:± Š¾æÔìÄîØ% disk./.usageGB‚1110.2437553405762 disk.nvme0n1.in‚ 167.5390625 disk.dm-1.out‚ 178.98046875! disk.rbd2.out‚107.54541015625& proc.memory.availableMB‚ 949.734375 proc.cpu.threads‚17 network.sent‚ 1.3788482e+07 disk.sda3.in‚ 0.01171875* proc.memory.percent‚17.540550231933594 cpu‚25.339158183041963 disk.sda2.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚19247.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.50938997746478 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 178.98046875 disk.sda3.out‚ 178.98046875! disk.nvme0c0n1.in‚ 167.5390625# disk.nvme0n1.out‚19247.02734375 disk.sda.in‚ 0.01171875 disk.sda1.in‚0$ memory_percent‚94.20328140258789 disk.drbd1777.out‚0 network.recv‚ 1.728361e+06 disk.sda2.out‚0 disk.rbd1.out‚2.453125 disk.rbd2.in‚0 proc.memory.rssMB‚ 2873.84375 disk.dm-0.in‚ 167.5390625 disk.dm-0.out‚19247.02734375 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda1.out‚0âÜî!’­‚(:‡ ™¾æÔÐáÇÕ gpu.0.gpu‚49.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚92.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&²õG9®‚(:® ™¾æÔòöÆÙ cpu‚25.354866802823636 disk.dm-0.out‚19344.53515625 disk.rbd1.out‚2.453125 disk.sda.out‚ 179.53515625 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.3794069e+07 network.recv‚ 1.732347e+06% disk.nvme0c0n1.out‚19344.53515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ proc.memory.availableMB‚951.1875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 proc.memory.rssMB‚ 2873.84375* proc.memory.percent‚17.540550231933594 disk.nvme0n1.in‚ 167.5390625# disk.nvme0n1.out‚19344.53515625 disk.dm-0.in‚ 167.5390625 disk.dm-1.out‚ 179.53515625) disk./.usagePercent‚74.50938997746478 disk.sda3.out‚ 179.53515625 disk.rbd2.out‚107.6357421875% disk./.usageGB‚1110.2437553405762! disk.nvme0c0n1.in‚ 167.5390625 disk.drbd1777.in‚0 disk.rbd0.out‚0$ memory_percent‚94.19441223144531 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0X‘Ì’¯‚(:‡ ¨¾æÔ‚®ÐÕ gpu.0.gpu‚27.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹.sÕ9°‚(:® ¨¾æÔܱ Ø% disk./.usageGB‚1110.2424812316895 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚107.6474609375 disk.nvme0n1.in‚ 167.5390625 disk.dm-0.in‚ 167.5390625 disk.sda2.out‚0) disk./.usagePercent‚74.50930447094835 disk.rbd1.out‚2.453125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 180.1171875$ memory_percent‚94.19443607330322( proc.memory.availableMB‚ 951.18359375 cpu‚25.37002649009479 network.sent‚ 1.3800341e+07 disk.rbd0.out‚0% disk.nvme0c0n1.out‚19345.23046875 network.recv‚ 1.736815e+06 disk.sda.in‚ 0.01171875 disk.sda.out‚ 180.1171875 disk.sda3.out‚ 180.1171875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 167.5390625 proc.memory.rssMB‚ 2873.84375* proc.memory.percent‚17.540550231933594 disk.rbd1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚19345.23046875 disk.dm-0.out‚19345.23046875ž”"2±- _wandbruntime‚4742  _runtime‚4742&ÎhL!²ÂT eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚351813224029696 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚351813206026752 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5065899067392 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5315455141376 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202181866 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚389477185692160 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202197845 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚351813224029696 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚28991740 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143478272 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚389477042213888 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 114196905 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚80338874 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚356879105094144 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚88000938 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚88000940 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚356879105094144 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5066391897600 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 114196905 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚342589495399424 _step‚492 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5065899067392 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5047739740756 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51347134 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚351813206026752 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚88000940 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚356879615927296 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚384161611012608 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 114196905 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚342589495399424 _step‚492 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5065899067392 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5047739740756 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚51347134 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚351813206026752 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚88000940 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚356879615927296 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚384161611012608 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 114196905 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚80338874 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚356879105094144 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚88000938 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚88000940 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚356879105094144 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5066391897600 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872ìH _ï2´- _wandbruntime‚4743  _runtime‚4743ͱpäR)µÌR BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80349799 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚351850036055552 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884526515195806e+09 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚88011746 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚88011744 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚351850054058496 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚356916811082752 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5066264194048 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5315820268032 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5048103350164 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚51353636 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚88011744 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5066264194048 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚28996163 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚356916811082752 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚351850054058496 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚389515283074048 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚384199486746112 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 trainer/global_step‚6044 _runtime‚4743.074463434 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚51353638 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚347672892509076 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202199974 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 114204207 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944UÀÝß#¶ÑR WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚51353636 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚88011744 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5066757024256 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5066264194048 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚28996163 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚342624789158912 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚356916811082752 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚351850054058496 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚389515283074048 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚384199486746112 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 trainer/global_step‚6044 _runtime‚4743.074463434 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚51353638 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚347672892509076 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202199974 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚159 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 114204207 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80349799 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚351850036055552 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884526515195806e+09 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚356916300249600 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current(#æïG‚ 868220928 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚88011746 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚88011744 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚351850054058496 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚356916811082752 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5066264194048 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5315820268032 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5048103350164 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5047612435772 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430í_…1“·‚(:ˆ ·¾æÔíÒ­Ö gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚161.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 q…a-¸‚(:¢ ·¾æÔ¯–ã×) disk./.usagePercent‚74.50937103290724 disk.rbd1.in‚0 disk.nvme0n1.in‚ 167.5390625 disk.drbd1777.in‚0 disk.rbd3.out‚0 network.recv‚ 1.740555e+06% disk./.usageGB‚1110.2434730529785 disk.sda3.out‚ 180.85546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚109.2841796875 disk.nvme0c0n1.out‚ 19442.875 disk.dm-0.in‚ 167.5390625 disk.dm-0.out‚ 19442.875 disk.sda.out‚ 180.85546875$ memory_percent‚94.19481754302979 disk.nvme0n1.out‚ 19442.875 network.sent‚ 1.3805546e+07 disk.sda1.out‚0( proc.memory.availableMB‚ 951.12109375* proc.memory.percent‚17.540550231933594 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd1.out‚2.453125 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 167.5390625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2873.84375 disk.rbd2.in‚0 disk.dm-1.out‚ 180.85546875 disk.rbd3.in‚0 cpu‚25.38471584669929 proc.cpu.threads‚17õr’z2¹- _wandbruntime‚4755  _runtime‚4755X¡«¤sº¤¨ :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28729572892189026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41097888350486755 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5123540163040161 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4253179132938385 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9870055317878723 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.43518519401550293 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9808720946311951 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5726878643035889 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.569786548614502 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2869086265563965 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010733182542026043 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3559625744819641 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5054767727851868 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26958051323890686 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9701389670372009 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_6_of_6‚1 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950621724128723 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.973099410533905 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4873654544353485 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882993698120117 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.035663895308971405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4318181872367859 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_2_of_3‚0.23744967579841614 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5466184020042419 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16465872526168823 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18533049523830414 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38661444187164307 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4785834848880768 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.556198239326477 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6365540027618408 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22551190853118896 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222983837128 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5452507138252258 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.44473183155059814 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4381091892719269 -jet/accuracy_4_of_4‚0.0063833934254944324 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5908154249191284 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5542052388191223 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.50637286901474 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16677811741828918 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8629640340805054 particle/accuracy_3_of_3‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9504618048667908 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.143833726644516 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29255974292755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10596708208322525 +particle/sensitivity‚0.3121536672115326 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4665239453315735 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9837164282798767 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24662162363529205 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852693676948547 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09648611396551132 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38407739996910095 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5057507753372192 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 _runtime‚4755.503485973 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17220188677310944 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_6_of_7‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17096298933029175 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16830584406852722 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2943950891494751 *jet/accuracy_1_of_5‚0.9685803651809692 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.226666659116745 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4862121343612671 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48571592569351196 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05975719168782234 particle/accuracy_5_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5737711191177368 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24428610503673553 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47503307461738586 jet/accuracy_5_of_7‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16641788184642792 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4021885395050049 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15467022359371185 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5192693471908569 *jet/accuracy_1_of_6‚0.9807692170143127 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.539699912071228 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_5_of_5‚0.001792114693671465 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 jet/accuracy_6_of_7‚0 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_5‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5793846249580383 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1529514044523239 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.45826295018196106 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38402819633483887 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9771800637245178 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846794605255127 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 jet/accuracy_2_of_7‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884526639471092e+09 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5037792921066284 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.175307497382164 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4067632257938385 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14905641973018646 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994708895683289 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19604171812534332 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5360453128814697 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02599063701927662 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1635701209306717 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03459189832210541 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22126325964927673 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38482096791267395 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1762503832578659 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9760881066322327 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36666664481163025 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.013216845691204071 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5004128217697144 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5030715465545654 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02526881732046604 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45716139674186707 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4955320954322815 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5502873659133911 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4075702130794525 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021600094623863697 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1654527336359024 +jet/accuracy_2_of_4‚0.39410072565078735 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.475694477558136 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21248380839824677 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5330356955528259 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 trainer/global_step‚6044 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9823828935623169 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 jet/accuracy_1_of_7‚1 jet/accuracy_7_of_7‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6088641285896301 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.567460298538208 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 epoch‚4 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3231481611728668 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6890572309494019 jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9123777747154236 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4884542226791382 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.43666666746139526 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4731508493423462 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34282246232032776 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20924176275730133 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9910252690315247 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5806319117546082 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10701382160186768 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1668270081281662 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13879312574863434 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8221355676651001 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38576388359069824 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996828556060791 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870483875274658 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22638046741485596 particle/accuracy_2_of_2‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24941444396972656 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47721531987190247 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31417012214660645 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4681558310985565 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11590038985013962 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4676806628704071 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44999998807907104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009637425653636456 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17096298933029175 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40174999833106995 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_6‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4918310344219208 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3866279125213623 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09613095223903656 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23232322931289673 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5829861164093018 particle/accuracy_5_of_7‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07083700597286224 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.409454345703125 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03648625314235687 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5186507701873779 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40375375747680664 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2650858461856842 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009410430677235126 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21814344823360443 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1502923220396042 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98186856508255 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5234200358390808 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.39503970742225647 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚494 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2643556296825409 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 5 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15184286236763 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960831999778748 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2708333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 (loss/val_total_loss‚44.5337141086556 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855532646179199 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.12900666892528534 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt4SRqqt¨†ú ‡ *FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18155105412006378 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15013541281223297 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990315854549408 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3892543911933899 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4036026895046234 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5825463533401489 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.530127227306366 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876269698143005 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2089267075061798 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17602497339248657 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023841483518481255 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9925520420074463 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 )jet/accuracy_1_of_2‚0.759566605091095 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028318881988525 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47584983706474304 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.511469304561615 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5054767727851868 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26250752806663513 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5218280553817749 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 particle/accuracy_2_of_6‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.489248126745224 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011612772941589 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2494252622127533 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4371459484100342 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16481180489063263 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10222554206848145 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23136870563030243 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5135641694068909 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4370899498462677 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4299529790878296 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012140020728111267 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09593716263771057 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018238980323076248 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20209155976772308 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.529321014881134 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07705970108509064 (particle/accuracy‚0.8507875204086304 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.347222238779068 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5092991590499878 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14772610366344452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029883047565817833 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08072695136070251 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1Bóúk_»©¨ jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9123777747154236 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4884542226791382 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.43666666746139526 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4731508493423462 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34282246232032776 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20924176275730133 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9910252690315247 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5806319117546082 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10701382160186768 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1668270081281662 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13879312574863434 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8221355676651001 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38576388359069824 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996828556060791 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870483875274658 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22638046741485596 particle/accuracy_2_of_2‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24941444396972656 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47721531987190247 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31417012214660645 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4681558310985565 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11590038985013962 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4676806628704071 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.44999998807907104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009637425653636456 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17096298933029175 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40174999833106995 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_6‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4918310344219208 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3866279125213623 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09613095223903656 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23232322931289673 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5829861164093018 particle/accuracy_5_of_7‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07083700597286224 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.409454345703125 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03648625314235687 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5186507701873779 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40375375747680664 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2650858461856842 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009410430677235126 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21814344823360443 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1502923220396042 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98186856508255 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5234200358390808 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.39503970742225647 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚494 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2643556296825409 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 5 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15184286236763 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960831999778748 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2708333432674408 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 (loss/val_total_loss‚44.5337141086556 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855532646179199 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.12900666892528534 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18155105412006378 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15013541281223297 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990315854549408 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3892543911933899 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4036026895046234 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5825463533401489 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.530127227306366 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876269698143005 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2089267075061798 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17602497339248657 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023841483518481255 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9925520420074463 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 )jet/accuracy_1_of_2‚0.759566605091095 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028318881988525 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47584983706474304 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.511469304561615 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5054767727851868 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26250752806663513 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5218280553817749 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 particle/accuracy_2_of_6‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.489248126745224 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011612772941589 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2494252622127533 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4371459484100342 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16481180489063263 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10222554206848145 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23136870563030243 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5135641694068909 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4370899498462677 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4299529790878296 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012140020728111267 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09593716263771057 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018238980323076248 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20209155976772308 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.529321014881134 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07705970108509064 (particle/accuracy‚0.8507875204086304 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.347222238779068 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5092991590499878 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14772610366344452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029883047565817833 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08072695136070251 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 particle/accuracy_3_of_7‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28729572892189026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41097888350486755 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5123540163040161 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4253179132938385 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9870055317878723 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.43518519401550293 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9808720946311951 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5726878643035889 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.569786548614502 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2869086265563965 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010733182542026043 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3559625744819641 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5054767727851868 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26958051323890686 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9701389670372009 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_6_of_6‚1 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950621724128723 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.973099410533905 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4873654544353485 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882993698120117 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.035663895308971405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4318181872367859 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_2_of_3‚0.23744967579841614 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5466184020042419 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16465872526168823 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18533049523830414 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38661444187164307 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4785834848880768 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.556198239326477 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6365540027618408 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22551190853118896 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12222222983837128 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5452507138252258 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.44473183155059814 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4381091892719269 -jet/accuracy_4_of_4‚0.0063833934254944324 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5908154249191284 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5542052388191223 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.50637286901474 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16677811741828918 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8629640340805054 particle/accuracy_3_of_3‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9504618048667908 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.143833726644516 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29255974292755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10596708208322525 +particle/sensitivity‚0.3121536672115326 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4665239453315735 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 PurityúÅÏ¥Å4/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9837164282798767 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026729561388492584 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24662162363529205 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852693676948547 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09648611396551132 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38407739996910095 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5057507753372192 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 _runtime‚4755.503485973 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17220188677310944 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_6_of_7‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17096298933029175 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16830584406852722 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2943950891494751 *jet/accuracy_1_of_5‚0.9685803651809692 6 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.226666659116745 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4862121343612671 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48571592569351196 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05975719168782234 particle/accuracy_5_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5737711191177368 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24428610503673553 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47503307461738586 jet/accuracy_5_of_7‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16641788184642792 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4021885395050049 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15467022359371185 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5192693471908569 *jet/accuracy_1_of_6‚0.9807692170143127 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.539699912071228 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_5_of_5‚0.001792114693671465 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 jet/accuracy_6_of_7‚0 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_4_of_5‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5793846249580383 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1529514044523239 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.45826295018196106 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38402819633483887 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9771800637245178 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846794605255127 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 jet/accuracy_2_of_7‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884526639471092e+09 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5037792921066284 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.175307497382164 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4067632257938385 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14905641973018646 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994708895683289 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19604171812534332 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5360453128814697 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02599063701927662 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1635701209306717 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03459189832210541 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22126325964927673 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38482096791267395 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1762503832578659 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9760881066322327 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36666664481163025 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.013216845691204071 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5004128217697144 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5030715465545654 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02526881732046604 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45716139674186707 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4955320954322815 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5502873659133911 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4075702130794525 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021600094623863697 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1654527336359024 +jet/accuracy_2_of_4‚0.39410072565078735 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.475694477558136 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21248380839824677 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5330356955528259 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 trainer/global_step‚6044 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9823828935623169 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 jet/accuracy_1_of_7‚1 jet/accuracy_7_of_7‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6088641285896301 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.567460298538208 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 epoch‚4 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3231481611728668 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6890572309494019îy *Š2¼- _wandbruntime‚4762  _runtime‚4762 mîŸÉ½Ã % _timestamp‚1.7884526714406612e+09 _runtime‚4762.995095623 _step‚495 ' lr-AdamW/pg1‚0.00010803770057910627 ' lr-AdamW/pg2‚0.00010803770057910627 trainer/global_step‚6049\³R°Î¾È ' lr-AdamW/pg1‚0.00010803770057910627 ' lr-AdamW/pg2‚0.00010803770057910627 trainer/global_step‚6049 % _timestamp‚1.7884526714406612e+09 _runtime‚4762.995095623 _step‚495ï ~³À2¿- _wandbruntime‚4762  _runtime‚4762ÛÔò–á'ÀÛO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343303207515136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5074787801600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114356521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 % _timestamp‚1.7884526714415233e+09Q-8.!ÁàO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5074787801600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114356521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 % _timestamp‚1.7884526714415233e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343303207515136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88208288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4763.670589697 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357685542491136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51479028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚497 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5056262912090 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884526721153479e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204327936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343358128685056À¢þð&ÄêM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884526721153479e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180387840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114374975 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204327936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88222304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343358128685056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚4763.670589697 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357685542491136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51479028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚497 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5074950866944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5056262912090 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStats÷+O¢Monitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115282573 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88909086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351144695859670 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5115117244928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355371890527744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5096281816534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4799.341306974 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360487007772672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115302611 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115282573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51879262 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290SvÏ)æ'ÑàO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29298675 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115282573 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88909086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚6099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351144695859670 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5115117244928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355371890527744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5096281816534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355371908530688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4799.341306974 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360487007772672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115302611 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115282573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51879262 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5096936254290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360487682138112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430ôQGéæžÒ‚(:“ ä¾æÔÌžþÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.464600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W”¬JÓ‚(:¿ ä¾æÔžÜèØ disk.sda2.in‚0 disk.rbd0.in‚0$ memory_percent‚91.29385948181152( proc.memory.availableMB‚ 1426.4140625 network.sent‚ 1.4065974e+07 disk.nvme0n1.in‚ 167.54296875 disk.rbd3.out‚0* proc.memory.percent‚17.549419403076172 network.recv‚ 1.757873e+06" disk.nvme0c0n1.in‚ 167.54296875 disk.rbd1.out‚2.453125 disk.rbd0.out‚0% disk./.usageGB‚1110.2437934875488 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 182.92578125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚20587.65771484375 disk.drbd1777.out‚0 disk.dm-1.out‚ 182.92578125 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 182.92578125& disk.nvme0n1.out‚20587.65771484375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2875.296875 cpu‚25.381375951044067 disk.dm-0.in‚ 167.54296875 disk.rbd2.out‚109.3623046875 proc.cpu.threads‚17) disk./.usagePercent‚74.50939253754014 disk.drbd1777.in‚0# disk.dm-0.out‚20587.65771484375 disk.rbd1.in‚0vKÎ2Ô- _wandbruntime‚4800  _runtime‚4800Øwùê&ÕäM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88923102 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346103335213056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884527084595737e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚6099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360544854618112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388082637913088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5367839585280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388082661853184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115301027 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88923102 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346103335213056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884527084595737e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚6099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitoD 5¡¬r.on_train_batch_end/allocation.large_pool.allocated‚88923104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360544854618112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5116593048576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388082637913088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5367839585280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388082661853184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091õDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29524419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116228663 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205818511 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5156103050752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81811637 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353930737727078 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358189542470144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5156103050752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116208625 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81812081 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358189560473088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358189560473088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353931409204194 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29523977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396513371499520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348793620571136 % _timestamp‚1.7884527443177223e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363345645520896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚4835.872353361 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363346319886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5137771593698 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89609886 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391103524058624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908ƒ°;°æ'ãàO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29524419 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116228663 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205818511 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5156103050752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81811637 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353930737727078 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358189542470144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5156103050752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116208625 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81812081 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358189560473088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358189560473088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353931409204194 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29523977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396513371499520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348793620571136 % _timestamp‚1.7884527443177223e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚363345645520896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚4835.872353361 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89609888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363346319886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5137771593698 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89609886 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391103524058624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚6149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5156759413248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52287662 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040øpÀ Ò2ä- _wandbruntime‚4836  _runtime‚4836ÈÙ‰>ZååM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5157578854400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396575692081152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89623904 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 _step‚505 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396575879631872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348848524701696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348848541741056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5157578854400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353986458292602 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚6149 _runtime‚4836.549791113 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5410680895488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884527449948804e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844Þ@Yð&æêM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5157578854400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138588028870 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396575692081152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89623904 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 _step‚505 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396575879631872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348848524701696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348848541741056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358245895508992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5157578854400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353986458292602 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚6149 _runtime‚4836.549791113 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5410680895488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884527449948804e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29749653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361007194412544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117134677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197745219072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚6199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366204957634560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.Yg›Qa small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394186084881920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 % _timestamp‚1.7884527803347857e+09 _runtime‚4871.889531365 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197745219072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚02Nj@æ'óàO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117134677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361007194412544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5197745219072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚6199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366204957634560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361007212415488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394186084881920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 % _timestamp‚1.7884527803347857e+09 _runtime‚4871.889531365 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197745219072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5197088856576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117154715 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90310686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82445715 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90310686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356716779594486 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29749653 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361007194412544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0üòò¦u2ô- _wandbruntime‚4872  _runtime‚4872M Œ¢ë&õåM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90324704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361063565454336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197908284416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117173195 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394247735619584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5178768930314 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884527810075595e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚4872.562622333 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117153131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361063565454336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚509 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29753702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351593748269056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394247711679488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90324702 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197908284416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351593731229696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361063547451392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5178768930314 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884527810075595e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366262130114560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚4872.562622333 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117153131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361063565454336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚509 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29753702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351593748269056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394247711679488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90324702 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197908284416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351593731229696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361063547451392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605ý†…??2÷- _wandbruntime‚4872  _runtime‚4872Ü;FIøÏ 0loss/FRt1/assignment_loss‚2.8272593251683493 0loss/FBt3/assÂn ignment_loss‚0.3310996219127749 1loss/SRqqt2/detection_loss‚1.1510675754398108 1loss/SRqqt1/assignment_loss‚1.451380324295976 1loss/FBt4/assignment_loss‚0.36414133589347486 0loss/SRqqt4/detection_loss‚2.769555861396449 epoch‚5 _step‚510 /loss/FRt3/assignment_loss‚3.223500152428945 .loss/FBt1/detection_loss‚2.923779010772705 trainer/global_step‚6199 _runtime‚ 4872.5637934 2loss/SRqqt2/assignment_loss‚1.3957013264640836 1loss/SRqqt3/detection_loss‚1.6005509194047725 &loss/total_loss‚46.056013888949906 2loss/SRqqt4/assignment_loss‚1.8436365907213517 .loss/FRt1/detection_loss‚2.438994171267206 /loss/FRt2/detection_loss‚1.8328516731659572 -loss/FBt3/detection_loss‚3.14033380150795 1loss/FBt2/assignment_loss‚0.23817677989287073 $ _timestamp‚1.788452781009498e+09 .loss/FRt2/assignment_loss‚2.03703254811904 /loss/FRt3/detection_loss‚2.0096023103185727 1loss/SRqqt1/detection_loss‚2.0194955583323133 -loss/FBt4/detection_loss‚2.92377222776413 0loss/FRt4/assignment_loss‚3.9188209663737905 2loss/SRqqt3/assignment_loss‚1.5387267092334442 1loss/FBt1/assignment_loss‚0.06186477706900665 /loss/FRt4/detection_loss‚3.2083878462964837 /loss/FBt2/detection_loss‚0.80628247571044739êpæZùÔ _runtime‚ 4872.5637934 2loss/SRqqt2/assignment_loss‚1.3957013264640836 1loss/SRqqt3/detection_loss‚1.6005509194047725 &loss/total_loss‚46.056013888949906 2loss/SRqqt4/assignment_loss‚1.8436365907213517 .loss/FRt1/detection_loss‚2.438994171267206 /loss/FRt2/detection_loss‚1.8328516731659572 -loss/FBt3/detection_loss‚3.14033380150795 1loss/FBt2/assignment_loss‚0.23817677989287073 $ _timestamp‚1.788452781009498e+09 .loss/FRt2/assignment_loss‚2.03703254811904 /loss/FRt3/detection_loss‚2.0096023103185727 1loss/SRqqt1/detection_loss‚2.0194955583323133 -loss/FBt4/detection_loss‚2.92377222776413 0loss/FRt4/assignment_loss‚3.9188209663737905 2loss/SRqqt3/assignment_loss‚1.5387267092334442 1loss/FBt1/assignment_loss‚0.06186477706900665 /loss/FRt4/detection_loss‚3.2083878462964837 /loss/FBt2/detection_loss‚0.8062824757104473 0loss/FRt1/assignment_loss‚2.8272593251683493 0loss/FBt3/assignment_loss‚0.3310996219127749 1loss/SRqqt2/detection_loss‚1.1510675754398108 1loss/SRqqt1/assignment_loss‚1.451380324295976 1loss/FBt4/assignment_loss‚0.36414133589347486 0loss/SRqqt4/detection_loss‚2.769555861396449 epoch‚5 _step‚510 /loss/FRt3/assignment_loss‚3.223500152428945 .loss/FBt1/detection_loss‚2.923779010772705 trainer/global_step‚6199þeÑÔR“ú‚(:ˆ ¯¿æÔŒ¥§× gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚101.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µBÌëEû‚(:º ¯¿æÔºÊÓØ disk.rbd2.in‚0! disk.rbd2.out‚111.35595703125 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚2.453125 disk.sda.out‚ 186.6015625& disk.nvme0n1.out‚20887.40673828125 cpu‚25.376357324864276 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 186.6015625* proc.memory.percent‚17.549419403076172 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 186.6015625 disk.rbd3.out‚0 disk.sda1.out‚0$ disk./.usageGB‚1110.244613647461) disk./.usagePercent‚74.50944757915998! disk.nvme0c0n1.in‚ 171.2421875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚91.30656719207764 network.recv‚ 1.783891e+06( disk.nvme0c0n1.out‚20887.40673828125 disk.nvme0n1.in‚ 171.2421875 disk.dm-1.in‚ 0.01171875) proc.memory.availableMB‚ 1424.33203125! proc.memory.rssMB‚ 2875.296875 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.dm-0.in‚ 171.2421875# disk.dm-0.out‚20887.40673828125 network.sent‚ 1.4381516e+07^Òeƒžü‚(:“ ¾¿æÔýó±Õ gpu.0.gpu‚41.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.630200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ųÎqGý‚(:¼ ¾¿æÔÇ¿Ô× disk.sda3.out‚ 187.51171875! disk.nvme0c0n1.in‚ 171.9609375 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 187.51171875 disk.sda1.in‚0! proc.memory.rssMB‚ 2875.296875* proc.memory.percent‚17.549419403076172 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚74.50945525938602! disk.rbd2.out‚111.35595703125( disk.nvme0c0n1.out‚20888.15673828125 disk.dm-0.in‚ 171.9609375 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚91.30692481994629( proc.memory.availableMB‚ 1424.2734375# disk.dm-0.out‚20888.15673828125 disk.rbd1.in‚0 cpu‚25.375321242581833 network.sent‚ 1.4490412e+07$ disk./.usageGB‚1110.244728088379 disk.rbd2.in‚0& disk.nvme0n1.out‚20888.15673828125 disk.sda2.in‚0 network.recv‚ 1.789967e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 171.9609375 disk.drbd1777.in‚0 disk.rbd1.out‚2.453125 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 187.51171875å„{¬“þ‚(:ˆ Í¿æÔáóÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö­úGÿ‚(:¼ Í¿æÔªšçØ$ disk./.usageGB‚1110.244728088379 disk.rbd0.in‚0 disk.drbd1777.out‚0# memory_percent‚91.3071870803833 disk.rbd3.in‚0! disk.rbd2.out‚111.43408203125 cpu‚25.374319215848463 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 network.sent‚ 1.4496046e+07 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda.in‚ 0.01171875! disk.nvme0c0n1.in‚ 171.9609375 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚20985.59814453125 disk.nvme0n1.in‚ 171.9609375 disk.dm-1.out‚ 187.99609375 disk.sda1.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚20985.59814453125 disk.drbd1777.in‚0 disk.sda2.in‚0 network.recv‚ 1.794524e+06 disk.sda3.out‚ 187.99609375 disk.rbd2.in‚0# disk.dm-0.out‚20985.59814453125! proc.memory.rssMB‚ 2875.296875) disk./.usagePercent‚74.50945525938602 disk.sda.out‚ 187.99609375) proc.memory.availableMB‚ 1424.23046875* proc.memory.percent‚17.549419403076172 disk.rbd3.out‚0 disk.dm-0.in‚ 171.9609375 disk.dm-1.in‚ 0.01171875æµâ2€- _wandbruntime‚4908  _runtime‚4908ƒ›ÓLÉà % _timestamp‚1.7884528164828823e+09 _runtime‚4908.037428056 _step‚511 ' lr-AdamW/pg1‚0.00010277317900528463 ' lr-AdamW/pg2‚0.00010277317900528463 trainer/global_step‚6249¹åé‹Î‚È _runtime‚4908.037428056 _step‚511 ' lr-AdamW/pg1‚0.00010277317900528463 ' lr-AdamW/pg2‚0.00010277317900528463 trainer/global_step‚6249 % _timestamp‚1.7884528164828823e+09ÿl‘,¨2ƒ- _wandbruntime‚4908  _runtime‚4908¨/dá'„ÛO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5219442272514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118080767 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5238731024896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91011488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402764281324544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363824864357888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5238731024896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5218787834758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91011486 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363824864357888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29974862 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91011488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402764095857664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354284033627136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884528164836955e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118060729 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5495498032640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118060729 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4908.038307203 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83078882 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397268621765120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209072215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397268597825024~{`æ'…àO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369063595382784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118060729 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5495498032640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118060729 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4908.038307203 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83078882 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397268621765120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209072215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397268597825024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5219442272514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118080767 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5238731024896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91011488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402764281324544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363824864357888 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5238731024896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5218787834758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91011486 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363824864357888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29974862 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91011488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402764095857664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354284033627136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884528164836955e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369062921017344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430€ð‡w2†- _wandbruntime‚4908  _runtime‚4908ac¯Ÿ‡åM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚513 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5220258707686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209124751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29979341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_tra%½ #Lin_batch_end/active.large_pool.allocated‚91025504 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397330272502784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚6249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397330248562688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚4908.729627674 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397330272502784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚6249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397330248562688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚4908.729627674 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363881217396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369120093484032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚513 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5220258707686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209124751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29979341 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354338937757696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91025504 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354338954797056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901¤ Ì2‰- _wandbruntime‚4908  _runtime‚4908žK@ZŠÔ 0loss/FRt4/assignment_loss‚3.9371248446404934 2loss/SRqqt1/assignment_loss‚1.5405526671219956 %loss/total_loss‚44.02388945075103 2loss/SRqqt3/assignment_loss‚1.7021695638321481 0loss/FBt4/assignment_loss‚0.5364723631313869 .loss/FRt3/detection_loss‚2.026399850845337 1loss/SRqqt1/detection_loss‚1.8765671140768312 trainer/global_step‚6249 /loss/FRt2/detection_loss‚1.6442565470333061 0loss/FRt1/assignment_loss‚2.7528893128037453 0loss/FRt2/assignment_loss‚1.8141552565559265 1loss/SRqqt4/detection_loss‚2.5954591155052187 _runtime‚4908.730689786 1loss/SRqqt3/detection_loss‚1.8093191818246301 /loss/FBt2/detection_loss‚0.5892513856218246 _step‚514 1loss/FBt2/assignment_loss‚0.22801518780183577 .loss/FRt4/detection_loss‚2.968529101461172 .loss/FBt4/detection_loss‚2.218014121055603 0loss/FBt3/assignment_loss‚0.4932218207977703 /loss/FRt1/detection_loss‚2.1279681340924332 0loss/FRt3/assignment_loss‚3.0193542941756872 1loss/SRqqt2/assignment_loss‚1.114134992448319 2loss/SRqqt4/assignment_loss‚1.8773635228474934 0loss/FBt1/assignment_loss‚0.3429269159847384 1loss/SRqqt2/detection_loss‚1.0975887181335373 -loss/FBt1/detection_loss‚3.05191433429718 /loss/FBt3/detection_loss‚2.6602411046624184 epoch‚5 % _timestamp‚1.7884528171764076e+09?Áͯ_‹Ù .loss/FBt4/detection_loss‚2.218014121055603 0loss/FBt3/assignment_loss‚0.4932218207977703 /loss/FRt1/detection_loss‚2.1279681340924332 0loss/FRt3/assignment_loss‚3.0193542941756872 1loss/SRqqt2/assignment_loss‚1.114134992448319 2loss/SRqqt4/assignment_loss‚1.8773635228474934 0loss/FBt1/assignment_loss‚0.3429269159847384 1loss/SRqqt2/detection_loss‚1.0975887181335373 -loss/FBt1/detection_loss‚3.05191433429718 /loss/FBt3/detection_loss‚2.6602411046624184 epoch‚5 % _timestamp‚1.7884528171764076e+09 0loss/FRt4/assignment_loss‚3.9371248446404934 2loss/SRqqt1/assignment_loss‚1.5405526671219956 %loss/total_loss‚44.02388945075103 2loss/SRqqt3/assignment_loss‚1.7021695638321481 0loss/FBt4/assignment_loss‚0.5364723631313869 .loss/FRt3/detection_loss‚2.026399850845337 1loss/SRqqt1/detection_loss‚1.8765671140768312 trainer/global_step‚6249 /loss/FRt2/detection_loss‚1.6442565470333061 0loss/FRt1/assignment_loss‚2.7528893128037453 0loss/FRt2/assignment_loss‚1.8141552565559265 1loss/SRqqt4/detection_loss‚2.5954591155052187 _runtime‚4908.730689786 1loss/SRqqt3/detection_loss‚1.8093191818246301 /loss/FBt2/detection_loss‚0.5892513856218246 _step‚514 1loss/FBt2/assignment_loss‚0.22801518780183577 .loss/FRt4/detection_loss‚2.968529101461172‚<¿ö*“Œ‚(:ˆ Ü¿æÔ¾óÖÕ gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LU?‚(:´ Ü¿æÔÕ¤—Ù) disk./.usagePercent‚74.50945781946136% disk./.usageGB‚1110.2447662353516 disk.dm-1.out‚ 188.5078125! disk.nvme0c0n1.in‚ 171.9609375 disk.drbd1777.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 188.5078125 disk.drbd1777.out‚0% proc.memory.availableMB‚ 1423.9375* proc.memory.percent‚17.549419403076172 network.sent‚ 1.4502299e+07 disk.rbd0.in‚0 disk.rbd1.out‚2.453125 disk.rbd3.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚20986.3955078125" disk.dm-0.out‚20986.3955078125 network.recv‚ 1.798831e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 171.9609375% disk.nvme0n1.out‚20986.3955078125 disk.sda1.out‚0 cpu‚25.373331713974782 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda3.out‚ 188.5078125 disk.rbd0.out‚0 disk.dm-0.in‚ 171.9609375 disk.rbd2.in‚0! disk.rbd2.out‚111.43408203125$ memory_percent‚91.30897521972656! proc.memory.rssMB‚ 2875.296875‚O5¡žŽ‚(:“ ë¿æÔáÎìÕ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.144199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÝôÄF‚(:» ë¿æÔÓŸù×! proc.memory.rssMB‚ 2875.296875 proc.cpu.threads‚17 disk.rbd0.in‚0) disk./.usagePercent‚74.50946447565725& disk.nvme0n1.out‚21083.88818359375 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0( disk.nvme0c0n1.out‚21083.88818359375 disk.drbd1777.out‚0 disk.dm-0.in‚ 171.9609375 disk.rbd1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.549419403076172 disk.rbd3.in‚0' proc.memory.availableMB‚ 1424.046875 disk.sda2.in‚0 disk.rbd1.out‚2.453125 network.sent‚ 1.4610359e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 171.9609375% disk./.usageGB‚1110.2448654174805 disk.drbd1777.in‚0 disk.dm-1.out‚ 189.26171875# disk.dm-0.out‚21083.88818359375! disk.nvme0c0n1.in‚ 171.9609375 network.recv‚ 1.804816e+06 disk.sda3.out‚ 189.26171875 disk.sda.out‚ 189.26171875 cpu‚25.37236707062107! disk.rbd2.out‚113.02392578125 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚91.30830764770508*¹?©2- _wandbruntime‚4943  _runtime‚4943×DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400351158648320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91712286 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366642516300288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91712288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91712288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚371922233131008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405889808456192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30199293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400351134708224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928b§Åâ•àO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362289534806418 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚366642516300288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91712288 SEDeviceStatsMonitor.on_train_batch_start/inactive_žÿ©H%split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91712288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚371922233131008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405889808456192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30199293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400351134708224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400351158648320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91712286 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366642498297344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119005235 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210731537 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚6299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5260439609130 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4944.616898551 _step‚517 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884528530617025e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366698851336192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91726302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064tµT6ð&˜êM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119005235 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210731537 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚6299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5260439609130 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚4944.616898551 _step‚517 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30203722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884528530617025e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53521028 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚366698851336192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91726302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0…È ø2™- _wandbruntime‚4944  _runtime‚4944NJBUšÏ /loss/FRt4/detection_loss‚3.0792965841293336 .loss/FBt3/detection_loss‚2.028354605038961 /loss/FRt1/assignment_loss‚2.880097253868977 0loss/SRqqt3/detection_loss‚1.549317338600241 /loss/FBt4/detection_loss‚2.5390375375747682 %loss/total_loss‚43.54703855129497 _runtime‚4944.618109433 0loss/FRt2/assignment_loss‚1.7955979290215864 1loss/SRqqt1/assignment_loss‚1.663875365726805 1loss/FBt2/assignment_loss‚0.20025371082276985 /loss/FRt4/assignment_loss‚4.208749647140503 2loss/SRqqt2/assignment_loss‚0.8844643357083385 0loss/FBt3/assignment_loss‚0.0726805672230613 1loss/FBt4/assignment_loss‚0.12175704436376691 trainer/global_step‚6299 # _timestamp‚1.78845285306375e+09 -loss/FRt1/detection_loss‚2.18236227085193 /loss/FRt2/detection_loss‚2.1267631185119567 /loss/FBt1/detection_loss‚3.1210495432217917 _step‚518 2loss/SRqqt3/assignment_loss‚1.4457499845724167 2loss/SRqqt4/assignment_loss‚1.9836638592409366 epoch‚5 0loss/FRt3/assignment_loss‚3.0959170245733416 1loss/SRqqt1/detection_loss‚1.9190861135721207 0loss/SRqqt2/detection_loss‚1.054651160850937 0loss/SRqqt4/detection_loss‚2.589074298287883 /loss/FBt2/detection_loss‚0.8274727205592972 0loss/FBt1/assignment_loss‚0.1258793187638112 .loss/FRt3/detection_loss‚2.051887219069434ºç9Z›Ô /loss/FRt1/assignment_loss‚2.880097253868977 0loss/SRqqt3/detection_loss‚1.549317338600241 /loss/FBt4/detection_loss‚2.5390375375747682 %loss/total_loss‚43.54703855129497 _runtime‚4944.618109433 0loss/FRt2/assignment_loss‚1.7955979290215864 1loss/SRqqt1/assignment_loss‚1.663875365726805 1loss/FBt2/assignment_loss‚0.20025371082276985 /loss/FRt4/assignment_loss‚4.208749647140503 2loss/SRqqt2/assignment_loss‚0.8844643357083385 0loss/FBt3/assignment_loss‚0.0726805672230613 1loss/FBt4/assignment_loss‚0.12175704436376691 trainer/global_step‚6299 # _timestamp‚1.78845285306375e+09 -loss/FRt1/detection_loss‚2.18236227085193 /loss/FRt2/detection_loss‚2.1267631185119567 /loss/FBt1/detection_loss‚3.1210495432217917 _step‚518 2loss/SRqqt3/assignment_loss‚1.4457499845724167 2loss/SRqqt4/assignment_loss‚1.9836638592409366 epoch‚5 0loss/FRt3/assignment_loss‚3.0959170245733416 1loss/SRqqt1/detection_loss‚1.9190861135721207 0loss/SRqqt2/detection_loss‚1.054651160850937 0loss/SRqqt4/detection_loss‚2.589074298287883 /loss/FBt2/detection_loss‚0.8274727205592972 0loss/FBt1/assignment_loss‚0.1258793187638112 .loss/FRt3/detection_loss‚2.051887219069434 /loss/FRt4/detection_loss‚3.0792965841293336 .loss/FBt3/detection_loss‚2.028354605038961†Ôžü“œ‚(:ˆ ú¿æÔíå±Õ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ûp<‚(:¬ ú¿æÔ†¼Ø disk.sda.out‚189.8125! disk`O$$û.nvme0c0n1.in‚ 171.9609375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1424.23828125! proc.memory.rssMB‚ 2875.296875 disk.sda3.out‚189.8125 network.recv‚ 1.809233e+06& disk.nvme0c0n1.out‚21181.876953125$ disk.nvme0n1.out‚21181.876953125 cpu‚25.371389762246768 disk.dm-0.in‚ 171.9609375$ memory_percent‚91.30713939666748 proc.cpu.threads‚17% disk./.usageGB‚1110.2449188232422 disk.drbd1777.in‚0* proc.memory.percent‚17.549419403076172 network.sent‚ 1.4616555e+07 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚21181.876953125 disk.dm-1.out‚189.8125 disk.rbd1.out‚2.453125! disk.rbd2.out‚113.02392578125 disk.nvme0n1.in‚ 171.9609375 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875) disk./.usagePercent‚74.50946805976272 disk.rbd1.in‚0 disk.rbd3.in‚0sVL?žž‚(:“ ‰ÀæÔïÜÃÕ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.006400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G„A?Ÿ‚(:´ ‰ÀæÔ”ä³Ø% disk./.usageGB‚1110.2444610595703 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 171.9609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚91.30809307098389! disk.nvme0c0n1.in‚ 171.9609375 cpu‚25.370424434890033 proc.cpu.threads‚17 network.sent‚ 1.4725152e+07 disk.sda3.out‚ 190.6015625& disk.nvme0c0n1.out‚21182.220703125 disk.sda1.out‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚21182.220703125 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875) proc.memory.availableMB‚ 1424.08203125 disk.rbd2.out‚113.1884765625 disk.rbd1.in‚0) disk./.usagePercent‚74.50943733885862 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 190.6015625 disk.sda.out‚ 190.6015625* proc.memory.percent‚17.549419403076172 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 171.9609375 disk.rbd1.out‚2.453125! proc.memory.rssMB‚ 2875.296875 network.recv‚ 1.815463e+06 disk.sda1.in‚0! disk.dm-0.out‚21182.220703125 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0bŠK³“ ‚(:ˆ ˜ÀæÔÂïÕ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð6H¡‚(:½ ˜ÀæÔë»éØ& disk.nvme0n1.out‚21280.88525390625* proc.memory.percent‚17.549419403076172 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 1.4731421e+07 disk.sda.out‚ 191.30078125) proc.memory.availableMB‚ 1423.83984375 cpu‚25.369509639581636) disk./.usagePercent‚74.50944297102437 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 191.30078125 disk.sda1.in‚0 disk.rbd2.out‚113.1884765625 disk.rbd1.out‚2.453125 disk.rbd2.in‚0# disk.dm-0.out‚21280.88525390625 proc.cpu.threads‚17 disk.rbd3.in‚0( disk.nvme0c0n1.out‚21280.88525390625 disk.nvme0n1.in‚ 171.9609375 disk.dm-0.in‚ 171.9609375! proc.memory.rssMB‚ 2875.296875 network.recv‚ 1.819793e+06 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 171.9609375 disk.drbd1777.out‚0$ memory_percent‚91.30957126617432 disk.drbd1777.in‚0% disk./.usageGB‚1110.2445449829102 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 191.30078125 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0:ÝQ³2¢- _wandbruntime‚4980  _runtime‚4980”~ñɣà _step‚519 ' lr-AdamW/pg1‚0.00010012652664689421 ' lr-AdamW/pg2‚0.00010012652664689421 trainer/global_step‚6349 % _timestamp‚1.7884528887350938e+09 _runtime‚4980.289478182ôo¤Î¤È trainer/global_step‚6349 % _timestamp‚1.7884528887350938e+09 _runtime‚4980.289478182 _step‚519 ' lr-AdamW/pg1‚0.00010012652664689421 ' lr-AdamW/pg2‚0.00010012652664689421‡Á;©2¥- _wandbruntime‚4980  _runtime‚4980QìËà'¦ÚO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212345959 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92413088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30425142 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4980.290393837 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374780870879232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374780870879232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359774446683136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84346404 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5320046274048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409014911360000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403433695531520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119912833 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212325919 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369460168242688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 )7å'§ßO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92413086 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212345959 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92413088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30425142 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚4980.290393837 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374780870879232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374780870879232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359774446683136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84346404 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5320046274048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409014911360000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403433695531520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119912833 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212325919 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369460168242688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788452889408533e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30429179 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4980.963556822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403495322329088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚6349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414Ð'¹fï&ªéM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84359051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30429621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369516503278592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788452889408533e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359829367853056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92427104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30429179 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚4980.963556822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403495322329088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚6349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430‰7Ÿî2«- _wandbruntime‚4980  _runtime‚4980*hùi\¬Ö _step‚522 1loss/FBt2/assignment_loss‚0.08814240148238592 /loss/FBt3/detection_loss‚2.3826811638745395 %loss/total_loss‚43.77187920447838 epoch‚5 0loss/FRt1/assignment_loss‚3.1843408726631326 /loss/FRt2/detection_loss‚1.6028624400496483 2loss/SRqqt4/assignment_loss‚1.9664162776122491 1loss/FBt4/assignment_loss‚0.16878160340711473 trainer/global_step‚6349 $ _timestamp‚1.788452889410607e+09 /loss/FRt2/assignment_loss‚1.672860967701879 /loss/FRt3/assignment_loss‚3.360259829709927 2loss/SRqqt1/assignment_loss‚1.6347925824691252 .loss/FRt3/detection_loss‚2.431726067637404 1loss/SRqqt2/detection_loss‚1.0278404143735254 /loss/FBt1/detection_loss‚2.6671533823013305 _runtime‚4980.964875177 .loss/FRt1/detection_loss‚2.386732133462074 .loss/FRt4/assignment_loss‚4.04658636365618 2loss/SRqqt2/assignment_loss‚1.1373719938966478 2loss/SRqqt3/assignment_loss‚1.7106385025775657 1loss/FBt3/assignment_loss‚0.24647825605511156 1loss/SRqqt4/detection_loss‚2.4554929468366833 1loss/FBt1/assignment_loss‚0.12576847086814952 /loss/FBt2/detection_loss‚0.6633764377588199 /loss/FBt4/detection_loss‚2.7764601051807403 /loss/FRt4/detection_loss‚2.8667109744889396 1loss/SRqqt1/detection_loss‚1.5728198253902896 1loss/SRqqt3/detection_loss‚1.5955851910249241\+a­Û %loss/total_loss‚43.77187920447838 epoch‚5 0loss/FRt1/assignment_loss‚3.1843408726631326 /loss/FRt2/detection_loss‚1.6028624400496483 2loss/SRqqt4/assignment_loss‚1.9664162776122491 1loss/FBt4/assignment_loss‚0.16878160340711473 trainer/global_step‚6349 $ _timestamp‚1.788452889410607e+09 /loss/FRt2/assignment_loss‚1.672860967701879 /loss/FRt3/assignment_loss‚3.360259829709927 2loss/SRqqt1/assignment_loss‚1.6347925824691252 .loss/FRt3/detection_loss‚2.431726067637404 1loss/SRqqt2/detection_loss‚1.0278404143735254 /loss/FBt1/detection_loss‚2.6671533823013305 _runtime‚4980.964875177 .loss/FRt1/detection_loss‚2.386732133462074 .loss/FRt4/assignment_loss‚4.04658636365618 2loss/SRqqt2/assignment_loss‚1.1373719938966478 2loss/SRqqt3/assignment_loss‚1.7106385025775657 1loss/FBt3/assignment_loss‚0.24647825605511156 1loss/SRqqt4/detection_loss‚2.4554929468366833 1loss/FBt1/assignment_loss‚0.12576847086814952 /loss/FBt2/detection_loss‚0.6633764377588199 /loss/FBt4/detection_loss‚2.7764601051807403 /loss/FRt4/detection_loss‚2.8667109744889396 1loss/SRqqt1/detection_loss‚1.5728198253902896 1loss/SRqqt3/detection_loss‚1.5955851910249241 _step‚522 1loss/FBt2/assignment_loss‚0.08814240148238592 /loss/FBt3/detection_loss‚2.3826811638745395Š e\‘®‚(:† §ÀæÔ¾âÕ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê´;¯‚(:° §ÀæÔºê¿Ø$ disk./.usageGB‚1110.241870880127 disk.rbd0.out‚0 disk.rbd2.out‚113.2626953125 disk.sda2.in‚0 network.recv‚ 1.823465e+06& disk.nvme0c0n1.out‚21380.142578125$ disk.nvme0n1.out‚21380.142578125 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda.out‚ 192.3046875 proc.cpu.threads‚17) disk./.usagePercent‚74.50926350974288 disk.sda1.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1423.828125 disk.sda3.out‚ 192.3046875! disk.nvme0c0n1.in‚ 171.9609375 disk.rbd1.out‚2.453125$ memory_percent‚91.30964279174805* proc.memory.percent‚17.549419403076172 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875! proc.memory.rssMB‚ 2875.296875 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 171.9609375 disk.drbd1777.out‚0 disk.dm-0.in‚ 171.9609375! disk.dm-0.out‚21380.142578125 cpu‚25.36850874806372 network.sent‚ 1.4736642e+07 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 192.3046875 disk.rbd3.out‚0 disk.rbd1.in‚0áÇG6ž°‚(:“ ¶ÀæÔÛ ¯Õ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚147.391' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.478199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÓ‹F±‚(:» ¶ÀæÔÈñÃ× disk.rbd3.out‚0 disk.sda.out‚ 192.94921875 disk.sda3.out‚ 192.94921875( disk.nvme0c0n1.out‚21383.34521484375 disk.dm-1.in‚ 0.01171875( disk./.usagePercent‚74.5093346798374 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 174.8828125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 174.8828125 disk.dm-1.out‚ 192.94921875& disk.nvme0n1.out‚21383.34521484375 disk.sda2.out‚0$ memory_percent‚91.30911827087402 cpu‚25.367520533761116 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.rbd0.out‚0( proc.memory.availableMB‚ 1423.9140625 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚114.7744140625! disk.nvme0c0n1.in‚ 174.8828125# disk.dm-0.out‚21383.34521484375 network.recv‚ 1.828912e+06% disk./.usageGB‚1110.2429313659668 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.drbd1777.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2875.296875* proc.memory.percent‚17.549419403076172 network.sent‚ 1.4844805e+07 disk.rbd1.in‚0øü1¯2²- _wandbruntime‚5016  _runtime‚50160edzÁ _runtime‚5016.069466189 _step‚523 & lr-AdamW/pg1‚9.880056659011656e-05 & lr-AdamW/pg2‚9.880056659011656e-05 trainer/global_step‚6399 % _timestamp‚1.7884529245150087e+09š,<"Ì´Æ & lr-AdamW/pg1‚9.880056659011656e-05 & lr-AdamW/pg2‚9.880056659011656e-05 trainer/global_step‚6399 % _timestamp‚1.7884529245150087e+09 _runtime‚5016.069466189 _step‚523‹=[µ2µ- _wandbruntime‚5016  _runtime‚5016RÁ0à'¶ÚO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406516232414720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93113886 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84980059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54329662 _step‚524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5341948290738 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372277802182144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6399 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412140372476928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406516208474624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120838885 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84979616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377639508627456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788452924515743e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5016.070354623 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93113888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93113888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6399 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412140372476928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406516208474624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120838885 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactißäve_split.large_pool.freed‚54329660 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84979616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120858923 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377639508627456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788452924515743e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5016.070354623 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93113888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362519653211136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93113888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412202875587584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361851507712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5625016375296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120877403 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93127902 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 `Xžð&ºêM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5625016375296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54337830 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120877403 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93127902 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884529252011833e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377696006728704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93127902 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367917339106966 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362574574381056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625184180224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406577859212288 _runtime‚5016.756040744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84992690 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372334155220992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412202875587584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361851507712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0õ¾Ž2»- _wandbruntime‚5016  _runtime‚5016Ê Ñ"[¼Õ 0loss/FRt3/assignment_loss‚3.5368716645808447 1loss/SRqqt2/detection_loss‚0.8139281411693521 %loss/total_loss‚42.96062951316908 % _timestamp‚1.7884529252028463e+09 .loss/FBt1/detection_loss‚2.776124656200409 /loss/FBt3/detection_loss‚2.2514613519112268 epoch‚5 _step‚526 2loss/SRqqt2/assignment_loss‚0.8582379764124547 2loss/SRqqt4/assignment_loss‚1.7788484548368761 1loss/FBt1/assignment_loss‚0.21518147976166802 _runtime‚5016.757151859 1loss/FBt2/assignment_loss‚0.19481925870741734 1loss/SRqqt4/detection_loss‚2.5190673643542874 0loss/FBt4/assignment_loss‚0.4453854808525648 .loss/FRt2/detection_loss‚1.736633284447285 .loss/FRt3/detection_loss‚2.194965057429813 0loss/SRqqt1/detection_loss‚1.986945552751422 /loss/FBt2/detection_loss‚0.6075259144852746 /loss/FBt4/detection_loss‚2.6630342230200768 1loss/SRqqt1/assignment_loss‚1.757272883822831 2loss/SRqqt3/assignment_loss‚1.8520385537772102 1loss/SRqqt3/detection_loss‚1.9067899196388873 trainer/global_step‚6399 /loss/FRt4/assignment_loss‚2.884418798344476 1loss/FBt3/assignment_loss‚0.42562377367479104 .loss/FRt1/detection_loss‚2.490576888973776 0loss/FRt1/assignment_loss‚2.6573265038038554 0loss/FRt2/assignment_loss‚1.4389494749215932 /loss/FRt4/detection_loss‚2.9686028552906856q|Ù`½Ú 1loss/FBt2/assignment_loss‚0.19481925870741734 1loss/SRqqt4/detection_loss‚2.5190673643542874 0loss/FBt4/assignment_loss‚0.4453854808525648 .loss/FRt2/detection_loss‚1.736633284447285 .loss/FRt3/detection_loss‚2.194965057429813 0loss/SRqqt1/detection_loss‚1.986945552751422 /loss/FBt2/detection_loss‚0.6075259144852746 /loss/FBt4/detection_loss‚2.6630342230200768 1loss/SRqqt1/assignment_loss‚1.757272883822831 2loss/SRqqt3/assignment_loss‚1.8520385537772102 1loss/SRqqt3/detection_loss‚1.9067899196388873 trainer/global_step‚6399 /loss/FRt4/assignment_loss‚2.884418798344476 1loss/FBt3/assignment_loss‚0.42562377367479104 .loss/FRt1/detection_loss‚2.490576888973776 0loss/FRt1/assignment_loss‚2.6573265038038554 0loss/FRt2/assignment_loss‚1.4389494749215932 /loss/FRt4/detection_loss‚2.9686028552906856 0loss/FRt3/assignment_loss‚3.5368716645808447 1loss/SRqqt2/detection_loss‚0.8139281411693521 %loss/total_loss‚42.96062951316908 % _timestamp‚1.7884529252028463e+09 .loss/FBt1/detection_loss‚2.776124656200409 /loss/FBt3/detection_loss‚2.2514613519112268 epoch‚5 _step‚526 2loss/SRqqt2/assignment_loss‚0.8582379764124547 2loss/SRqqt4/assignment_loss‚1.7788484548368761 1loss/FBt1/assignment_loss‚0.21518147976166802 _runtime‚5016.757151859ŽŽ}“¾‚(:ˆ ÅÀæÔŸñßÕ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jHë4¿‚(:© ÅÀæÔÉ·Ù disk.rbd2.out‚114.9150390625 disk.dm-0.in‚ 174.890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚2.453125 disk.rbd3.out‚0 proc.memory.availableMB‚1424! proc.memory.rssMB‚ 2875.296875 network.recv‚ 1.833012e+06$ disk./.usageGB‚1110.242977142334 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚21480.88330078125# disk.dm-0.out‚21480.88330078125 disk.rbd1.in‚0 disk.sda.out‚ 193.55078125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 174.890625 disk.nvme0n1.in‚ 174.890625& disk.nvme0n1.out‚21480.88330078125 disk.drbd1777.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 193.55078125 disk.sda3.out‚ 193.55078125 disk.rbd0.in‚0* proc.memory.percent‚17.549419403076172 cpu‚25.366557675459678 disk.sda1.out‚0 network.sent‚ 1.4850324e+07( disk./.usagePercent‚74.5093377519278 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 memory_percent‚ 91.30859375ƒ&Ÿ[žÀ‚(:“ ÔÀæÔèó­Õ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚147.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ðNAÁ‚(:¶ ÔÀæÔü—ïØ disk.dm-1.out‚ 194.19140625 cpu‚25.365626240272903 proc.cpu.threads‚17$ disk./.usageGB‚1110.243278503418 disk.rbd0.in‚0' disk.nvme0c0n1.out‚21482.8212890625 network.sent‚ 1.4959345e+07' disk./.usagePercent‚74.509357976523 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 194.19140625 disk.sda1.out‚0 disk.sda3.out‚ 194.19140625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 175.9609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚21482.8212890625( proc.memory.availableMB‚ 1423.7578125! proc.memory.rssMB‚ 2875.296875 disk.rbd1.in‚0 disk.dm-0.in‚ 175.9609375$ memory_percent‚91.31007194519043* proc.memory.percent‚17.549419403076172 disk.rbd1.out‚2.453125 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 175.9609375 network.recv‚ 1.839347e+06 disk.rbd2.out‚114.9150390625% disk.nvme0n1.out‚21482.8212890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0 disk.rbd0.out‚0ö ùí2Â- _wandbruntime‚5052  _runtime‚5052îØ“ÆÃÀ trainer/global_step‚6449 $ _timestamp‚1.788452960611149e+09 _runtime‚5052.165576917 _step‚527 & lr-AdamW/pg1‚9.747321893681542e-05 & lr-AdamW/pg2‚9.747321893681542e-05éD'JËÄÅ & lr-AdamW/pg1‚9.747321893681542e-05 & lr-AdamW/pg2‚9.747321893681542e-05 trainer/global_step‚6449 $ _timestamp‚1.788452960611149e+09 _runtime‚5052.165576917 _step‚527 Ê“÷2Å- _wandbruntime‚5052  _runtime‚5052lŒª^ÆÛO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884529606119175e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884529606119175e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 J߉2È- _wandbruntime‚5052  _runtime‚5052»¼¦@ê&ÉäM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6449 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415328530833920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚529 _runtime‚ 5052.85185148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54746228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370703380974374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121783391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375151807163392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370702709497050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5403493689344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121803455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30880998 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365319763869696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668110798336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚415328341186048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5383600065318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93828702 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884529612969234e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5382945627354 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093Û´ lX ÊéM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121783391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375151807163392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370702709497050 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5403493689344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375151825166336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121803455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30880998 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365319763869696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668110798336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚415328341186048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5383600065318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93828702 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93828702 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884529612969234e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5382945627354 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365319780909056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6449 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380555318855680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415328530833920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚529 _runtime‚ 5052.85185148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54746228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370703380974374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336‘ûIFÃ2Ë- _wandbruntime‚5052  _runtime‚5052¨w0„[ÌÕ /loss/FRt1/detection_loss‚2.5593243764667974 .loss/FRt4/detection_loss‚3.172191029009612 trainer/global_step‚6449 2loss/SRqqt2/assignment_loss‚1.1286439041522416 0loss/FBt2/assignment_loss‚0.2238566886380202 1loss/SRqqt4/detection_loss‚2.9747493023222145 /loss/FBt2/detection_loss‚0.6740445382971442 %loss/total_loss‚44.62599372709212 epoch‚5 0loss/FRt1/assignment_loss‚3.4085564046371273 /loss/FBt3/detection_loss‚1.9451920847098032 1loss/FBt1/assignment_loss‚0.24398989402900703 /loss/FRt3/detection_loss‚2.1427355074882506 /loss/FBt4/detection_loss‚2.2070457296712056 _step‚530 % _timestamp‚1.7884529612986894e+09 2loss/SRqqt3/assignment_loss‚1.2953800036009246 /loss/FRt2/detection_loss‚1.6416662543736122 0loss/SRqqt2/detection_loss‚1.306007700616663 1loss/SRqqt3/detection_loss‚1.7493868360916773 /loss/FRt3/assignment_loss‚2.450855926275253 1loss/SRqqt1/detection_loss‚1.9268164037072912 _runtime‚5052.852948398 /loss/FRt4/assignment_loss‚4.328355042830758 2loss/SRqqt1/assignment_loss‚1.3788888116386977 2loss/SRqqt4/assignment_loss‚2.1916737190701743 .loss/FBt1/detection_loss‚2.845075845718384 0loss/FRt2/assignment_loss‚1.5507155248412379 0loss/FBt3/assignment_loss‚0.6364109830620369 0loss/FBt4/assignment_loss‚0.6444312158439841g™7º`ÍÚ /loss/FRt3/assignment_loss‚2.450855926275253 1loss/SRqqt1/detection_loss‚1.9268164037072912 _runtime‚5052.852948398 /loss/FRt4/assignment_loss‚4.328355042830758 2loss/SRqqt1/assignment_loss‚1.3788888116386977 2loss/SRqqt4/assignment_loss‚2.1916737190701743 .loss/FBt1/detection_loss‚2.845075845718384 0loss/FRt2/assignment_loss‚1.5507155248412379 0loss/FBt3/assignment_loss‚0.6364109830620369 0loss/FBt4/assignment_loss‚0.6444312158439841 /loss/FRt1/detection_loss‚2.5593243764667974 .loss/FRt4/detection_loss‚3.172191029009612 trainer/global_step‚6449 2loss/SRqqt2/assignment_loss‚1.1286439041522416 0loss/FBt2/assignment_loss‚0.2238566886380202 1loss/SRqqt4/detection_loss‚2.9747493023222145 /loss/FBt2/detection_loss‚0.6740445382971442 %loss/total_loss‚44.62599372709212 epoch‚5 0loss/FRt1/assignment_loss‚3.4085564046371273 /loss/FBt3/detection_loss‚1.9451920847098032 1loss/FBt1/assignment_loss‚0.24398989402900703 /loss/FRt3/detection_loss‚2.1427355074882506 /loss/FBt4/detection_loss‚2.2070457296712056 _step‚530 % _timestamp‚1.7884529612986894e+09 2loss/SRqqt3/assignment_loss‚1.2953800036009246 /loss/FRt2/detection_loss‚1.6416662543736122 0loss/SRqqt2/detection_loss‚1.306007700616663 1loss/SRqqt3/detection_loss‚1.7493868360916773’X {¶’΂(:‡ ãÀæÔŒÂïÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚99.948' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒøFÏ‚(:» ãÀæÔλ®Ù disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda1.in‚0 disk.sda3.out‚ 194.67578125! disk.rbd2.out‚115.02490234375 network.sent‚ 1.4964552e+07% disk./.usageGB‚1110.2430877685547# disk.dm-0.out‚21580.50830078125 disk.sda.out‚ 194.67578125 disk.rbd2.in‚0& disk.nvme0n1.out‚21580.50830078125 network.recv‚ 1.843062e+06 disk.drbd1777.in‚0 disk.dm-0.in‚ 175.9609375 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.36463996657673 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 175.9609375! proc.memory.rssMB‚ 2875.296875 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875! disk.nvme0c0n1.in‚ 175.9609375$ memory_percent‚91.30854606628418( proc.memory.availableMB‚ 1424.0078125 proc.cpu.threads‚17( disk./.usagePercent‚74.5093451761463 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.dm-1.out‚ 194.67578125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚21580.50830078125* proc.memory.percent‚17.549419403076172cQžÐ‚(:“ òÀæÔ†ïÝÖ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.606' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.121199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tz¼ÖCÑ‚(:¸ òÀæÔ¢™èØ disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875" disk.dm-0.out‚21679.3447265625! disk.nvme0c0n1.in‚ 176.6796875 disk.dm-1.in‚ 0.01171875* proc.memory.percent‚17.549419403076172 proc.cpu.threads‚17% disk./.usageGB‚1110.2401657104492 disk.sda.in‚ 0.01171875% disk.nvme0n1.out‚21679.3447265625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125$ memory_percent‚94.20638084411621! proc.memory.rssMB‚ 2875.296875 cpu‚25.36376860203354 disk.sda3.out‚ 195.11328125' disk.nvme0c0n1.out‚21679.3447265625 disk.drbd1777.out‚0 network.sent‚ 1.5073628e+07 disk.sda.out‚ 195.11328125 disk.rbd0.in‚0' proc.memory.availableMB‚ 949.2265625) disk./.usagePercent‚74.50914907437506! disk.rbd2.out‚115.02490234375 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 176.6796875 disk.dm-0.in‚ 176.6796875 disk.dm-1.out‚ 195.11328125 network.recv‚ 1.849401e+06 disk.rbd3.in‚0Þ²òžÒ‚(:“ ÁæÔÇö»Õ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.512' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.902400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÁ8uBÓ‚(:· ÁæÔ¤¨žÚ% disk./.usageGB‚1110.2401657104492 disk.sda1.in‚0! disk.rbd2.out‚115.16552734375" disk.dm-0.out‚21679.6064453125) disk./.usagePercent‚74.50914907437506( proc.memory.availableMB‚ 950.76171875* proc.memory.percent‚17.549419403076172 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 176.6796875 disk.drbd1777.in‚0 cpu‚25.362783991825303' disk.nvme0c0n1.out‚21679.6064453125 disk.sda.out‚ 195.8046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 195.8046875 disk.sda3.out‚ 195.8046875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 176.6796875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.dm-0.in‚ 176.6796875$ memory_percent‚94.19701099395752 proc.cpu.threads‚17 disk.rbd3.out‚0 network.sent‚ 1.5079266e+07 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚21679.6064453125 disk.rbd1.out‚2.453125 network.recv‚ 1.853474e+06 disk.dm-1.in‚ 0.01171875! proc.memory.rssMB‚ 2875.296875]€ËJ2Ô- _wandbruntime‚5088  _runtime‚5088àÅ¡ÆÕÀ & lr-AdamW/pg1‚9.614476030425222e-05 & lr-AdamW/pg2‚9.614476030425222e-05 trainer/global_step‚6499 % _timestamp‚1.7884529966823354e+09 _runtime‚ 5088.23672459 _step‚531. Ÿ:ËÖÅ trainer/global_step‚6499 % _timestamp‚1.7884529966823354e+09 _runtime‚ 5088.23672459 _step‚531 & lr-AdamW/pg1‚9.614476030425222e-05 & lr-AdamW/pg2‚9.614476030425222e-05“)÷R2×- _wandbruntime‚5088  _runtime‚5088GlôÉá'ØÛO trainer/global_step‚6499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94515486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5088.237756596 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5443660054016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122711027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418391325721600 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412681282241024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55146460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5710043480576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872Ð1UÙàO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5088.237756596 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5443660054016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122711027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55146462 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418391325721600 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5422964531798 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412681282241024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_sta0^úI rt/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55146460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5710043480576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚6499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94515486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050”«~2Ú- _wandbruntime‚5088  _runtime‚5088à—í©ê&ÛäM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5711076189184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788452997363043e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5088.918084074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122729507 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5423780966762 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412742932978688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122709443 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5710920967168 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377969459105792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94529504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 trainer/global_step‚6499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064øãï&ÜéM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5711076189184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788452997363043e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94529502 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5088.918084074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122729507 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5423780966762 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412742932978688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122709443 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5710920967168 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94529502 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377969459105792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94529504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418454033107968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 trainer/global_step‚6499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122709443 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377969459105792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872•PÂtd2Ý- _wandbruntime‚5088  _runtime‚5088ÿíh^ÞØ epoch‚5 1loss/FBt2/assignment_loss‚0.17522823014125644 1loss/SRqqt3/detection_loss‚2.1267063312041454 _step‚534 0loss/FRt1/assignment_loss‚3.0633932426571846 0loss/FRt4/assignment_loss‚3.5095670728972466 1loss/FBt4/assignment_loss‚0.39434582284755176 .loss/FRt1/detection_loss‚2.138548804580101 .loss/FBt1/detection_loss‚2.933499291539192 trainer/global_step‚6499 1loss/SRqqt4/detection_loss‚2.5117080548117237 .loss/FBt3/detection_loss‚2.565908297896385 0loss/FRt3/assignment_loss‚3.7643351294100285 /loss/FRt2/detection_loss‚1.7179838501449143 1loss/SRqqt2/detection_loss‚1.0069245065467944 &loss/total_loss‚43.249698028647956 0loss/FRt2/assignment_loss‚1.2782905335937227 2loss/SRqqt2/assignment_loss‚0.9090726777533258 1loss/FBt1/assignment_loss‚0.38380638882517815 /loss/FRt4/detection_loss‚2.8341909502491807 1loss/SRqqt1/detection_loss‚1.5814726125252874 /loss/FBt2/detection_loss‚0.5998060199523267 _runtime‚5088.919154842 1loss/SRqqt1/assignment_loss‚1.489201592020037 0loss/FBt3/assignment_loss‚0.4498333897208795 .loss/FRt3/detection_loss‚2.072629443059365 /loss/FBt4/detection_loss‚2.3874326282077365 2loss/SRqqt4/assignment_loss‚1.8007221497114627 2loss/SRqqt3/assignment_loss‚1.5550910083529277 % _timestamp‚1.7884529973648262e+09•OðbcßÝ 2loss/SRqqt3/assignment_loss‚1.5550910083529277 % _timestamp‚1.7884529973648262e+09 epoch‚5 1loss/FBt2/assignment_loss‚0.17522823014125644 1loss/SRqqt3/detection_loss‚2.1267063312041454 _step‚534 0loss/FRt1/assignment_loss‚3.0633932426571846 0loss/FRt4/assignment_loss‚3.5095670728972466 1loss/FBt4/assignment_loss‚0.39434582284755176 .loss/FRt1/detection_loss‚2.138548804580101 .loss/FBt1/detection_loss‚2.933499291539192 trainer/global_step‚6499 1loss/SRqqt4/detection_loss‚2.5117080548117237 .loss/FBt3/detection_loss‚2.565908297896385 0loss/FRt3/assignment_loss‚3.7643351294100285 /loss/FRt2/detection_loss‚1.7179838501449143 1loss/SRqqt2/detection_loss‚1.0069245065467944 &loss/total_loss‚43.249698028647956 0loss/FRt2/assignment_loss‚1.2782905335937227 2loss/SRqqt2/assignment_loss‚0.9090726777533258 1loss/FBt1/assignment_loss‚0.38380638882517815 /loss/FRt4/detection_loss‚2.8341909502491807 1loss/SRqqt1/detection_loss‚1.5814726125252874 /loss/FBt2/detection_loss‚0.5998060199523267 _runtime‚5088.919154842 1loss/SRqqt1/assignment_loss‚1.489201592020037 0loss/FBt3/assignment_loss‚0.4498333897208795 .loss/FRt3/detection_loss‚2.072629443059365 /loss/FBt4/detection_loss‚2.3874326282077365 2loss/SRqqt4/assignment_loss‚1.8007221497114627–cÑÛžà‚(:“ ÁæÔ–‡âÕ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚135.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú~êGEá‚(:º ÁæÔÙ) disk./.usagePercent‚74.50914421023191 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 176.6796875 disk.sda3.out‚ 196.54296875# disk.dm-0.out‚21778.07763671875& proc.memory.availableMB‚ 950.484375 network.recv‚ 1.858175e+06 disk.sda1.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2875.296875 cpu‚25.36187913245715 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21778.07763671875 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1110.2400932312012& disk.nvme0n1.out‚21778.07763671875 disk.sda2.in‚0 disk.dm-1.out‚ 196.54296875* proc.memory.percent‚17.549419403076172 disk.rbd3.out‚0 network.sent‚ 1.5085516e+07 disk.sda1.in‚0! disk.rbd2.out‚116.67724609375 disk.dm-0.in‚ 176.6796875 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 196.54296875$ memory_percent‚94.19870376586914 disk.rbd0.in‚0 disk.nvme0n1.in‚ 176.6796875 disk.rbd1.in‚0û^殓â‚(:ˆ ŸÁæÔ€ÞòÕ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RD w?ã‚(:´ ŸÁæÔÚ£ÆØ disk.drbd1777.out‚0 disk.dm-0.in‚ 176.6796875 cpu‚25.360975940190613! disk.rbd2.out‚116.77099609375 disk.sda2.in‚0! proc.memory.rssMB‚ 2875.296875 network.recv‚ 1.864106e+06 disk.rbd3.in‚0 disk.sda.out‚ 197.109375( disk.nvme0c0n1.out‚21778.31982421875 disk.dm-1.in‚ 0.01171875 network.sent‚ 1.5193589e+07) disk./.usagePercent‚74.50914549026957 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 176.6796875# disk.dm-0.out‚21778.31982421875% disk./.usageGB‚1110.2401123046875 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 176.6796875 proc.cpu.threads‚17 disk.sda2.out‚0% proc.memory.availableMB‚ 950.46875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 197.109375 disk.sda.in‚ 0.01171875& disk.nvme0n1.out‚21778.31982421875 disk.drbd1777.in‚0 disk.dm-1.out‚ 197.109375$ memory_percent‚94.19879913330078* proc.memory.percent‚17.549419403076172 disk.rbd1.out‚2.453125 disk.rbd2.in‚0-}ÉÏ2ä- _wandbruntime‚5124  _runtime‚5124^\ËÆåÀ $ _timestamp‚1.788453032938505e+09 _runtime‚5124.492869697 _step‚535 & lr-AdamW/pg1‚9.481546754121488e-05 & lr-AdamW/pg2‚9.481546754121488e-05 trainer/global_step‚6549WLKJËæÅ _step‚535 & lr-AdamW/pg1‚9.481546754121488e-05 & lr-AdamW/pg2‚9.481546754121488e-05 trainer/global_step‚6549 $ _timestamp‚1.788453032938505e+09 _runtime‚5124.492869697—Y P…2ç- _wandbruntime‚5124  _runtime‚5124.hÈ]Ÿ èÚO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95216288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55554860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceSta6ÌÅAtsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚536 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5124.494562488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421516783155200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31330174 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5753136043520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376219744143458 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95216286 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376219744143458 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95216286 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚536 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5124.494562488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421516783155200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380730776012288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376219072666342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31330174 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5753136043520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041˜ õ|é2ê- _wandbruntime‚5125  _runtime‚5125=©îüë&ëåM trainer/global_step‚6549 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31335049 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5125.159781089 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5464616306170 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415825469861888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86898079 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421579298174976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5484808925184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884530336048179e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380787111048192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415825493801984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376274793231866 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5485465300992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380787129051136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386271919973376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370810176925696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380787111048192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123635495 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753828313088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95230302 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386272594352128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55563030 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964Tæ‡ÒšìêM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95230302 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386272594352128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55563030 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚6549 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31335049 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5125.159781089 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5464616306170 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415825469861888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86898079 §f7VXDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421579298174976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5484808925184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884530336048179e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380787111048192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415825493801984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376274793231866 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5485465300992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380787129051136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386271919973376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370810176925696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380787111048192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123635495 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753828313088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360™ð4ó2í- _wandbruntime‚5125  _runtime‚5125E ™[îÕ 0loss/FRt3/assignment_loss‚3.7662047797983345 2loss/SRqqt2/assignment_loss‚1.1974592798699935 /loss/FBt1/detection_loss‚2.4528598338365555 % _timestamp‚1.7884530336067104e+09 /loss/FBt3/detection_loss‚2.8365544279416404 epoch‚5 /loss/FRt1/assignment_loss‚2.887745224848026 /loss/FRt4/assignment_loss‚3.785752819134639 /loss/FRt4/detection_loss‚2.8937054184766917 1loss/SRqqt2/detection_loss‚0.9419941859263362 /loss/FBt2/detection_loss‚0.6460802448824285 2loss/SRqqt4/assignment_loss‚1.8001007795747783 /loss/FRt1/detection_loss‚2.3675096289413733 .loss/FBt4/detection_loss‚2.833866536617279 /loss/FRt2/assignment_loss‚1.999524575471878 2loss/SRqqt3/assignment_loss‚1.0880432487573735 _runtime‚5125.161050602 2loss/SRqqt1/assignment_loss‚1.2812219643644218 0loss/FBt1/assignment_loss‚0.4365581530340326 .loss/FRt2/detection_loss‚1.864980868808925 0loss/SRqqt4/detection_loss‚2.433625863658057 trainer/global_step‚6549 /loss/FRt3/detection_loss‚2.0962408394976095 %loss/total_loss‚43.98538862701962 _step‚538 1loss/FBt2/assignment_loss‚0.15379322553191138 1loss/FBt3/assignment_loss‚0.17782597983669904 0loss/FBt4/assignment_loss‚0.2394680616454126 1loss/SRqqt1/detection_loss‚1.7127633783324012 1loss/SRqqt3/detection_loss‚2.0915093082328173 Øü`ïÚ /loss/FBt3/detection_loss‚2.8365544279416404 epoch‚5 /loss/FRt1/assignment_loss‚2.887745224848026 /loss/FRt4/assignment_loss‚3.785752819134639 /loss/FRt4/detection_loss‚2.8937054184766917 1loss/SRqqt2/detection_loss‚0.9419941859263362 /loss/FBt2/detection_loss‚0.6460802448824285 2loss/SRqqt4/assignment_loss‚1.8001007795747783 /loss/FRt1/detection_loss‚2.3675096289413733 .loss/FBt4/detection_loss‚2.833866536617279 /loss/FRt2/assignment_loss‚1.999524575471878 2loss/SRqqt3/assignment_loss‚1.0880432487573735 _runtime‚5125.161050602 2loss/SRqqt1/assignment_loss‚1.2812219643644218 0loss/FBt1/assignment_loss‚0.4365581530340326 .loss/FRt2/detection_loss‚1.864980868808925 0loss/SRqqt4/detection_loss‚2.433625863658057 trainer/global_step‚6549 /loss/FRt3/detection_loss‚2.0962408394976095 %loss/total_loss‚43.98538862701962 _step‚538 1loss/FBt2/assignment_loss‚0.15379322553191138 1loss/FBt3/assignment_loss‚0.17782597983669904 0loss/FBt4/assignment_loss‚0.2394680616454126 1loss/SRqqt1/detection_loss‚1.7127633783324012 1loss/SRqqt3/detection_loss‚2.0915093082328173 0loss/FRt3/assignment_loss‚3.7662047797983345 2loss/SRqqt2/assignment_loss‚1.1974592798699935 /loss/FBt1/detection_loss‚2.4528598338365555 % _timestamp‚1.7884530336067104e+09šõʩГð‚(:ˆ ®ÁæÔͼå× gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Á¦þ:ñ‚(:¯ ®ÁæÔƒ«¥Ú disk.sda.out‚ 197.71875 disk.dm-1.in‚ 0.01171875 network.recv‚ 1.868447e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 197.71875$ memory_percent‚94.19763088226318( proc.memory.availableMB‚ 950.66015625 proc.cpu.threads‚17 disk.rbd1.out‚2.453125 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 176.6796875' disk.nvme0c0n1.out‚21876.3876953125 disk.sda2.in‚0% disk./.usageGB‚1110.2402153015137 disk.sda3.out‚ 197.71875! disk.rbd2.out‚116.77099609375 cpu‚25.360053322867397 disk.rbd3.out‚0 disk.nvme0n1.in‚ 176.6796875% disk.nvme0n1.out‚21876.3876953125 disk.drbd1777.in‚0' disk./.usagePercent‚74.509152402473 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0! proc.memory.rssMB‚ 2875.296875 network.sent‚ 1.5199733e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 176.6796875" disk.dm-0.out‚21876.3876953125* proc.memory.percent‚17.549419403076172 ”÷“ò‚(:ˆ ½ÁæÔ×ðÙÖ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øJAó‚(:¶ ½ÁæÔÇÇû×) disk./.usagePercent‚74.50915521855589 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 198.40234375 disk.rbd0.in‚0$ memory_percent‚94.19915676116943 network.sent‚ 1.53082e+07% disk./.usageGB‚1110.2402572631836 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 116.873046875% disk.nvme0n1.out‚21973.8427734375* proc.memory.percent‚17.549419403076172 network.recv‚ 1.874706e+06" disk.dm-0.out‚21973.8427734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda3.in‚ 0.01171875 disk.dm-1.in‚ 0.01171875 disk.drbd1777.in‚0 disk.dm-1.out‚ 198.40234375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 198.40234375! disk.nvme0c0n1.in‚ 176.6796875 disk.drbd1777.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2875.296875 cpu‚25.359177613813067 proc.cpu.threads‚17 disk.dm-0.in‚ 176.6796875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚21973.8427734375 disk.nvme0n1.in‚ 176.6796875( proc.memory.availableMB‚ 950.41015625Ÿ™õ ’ô‚(:‡ ÌÁæÔ¼ŽÄÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äs&Dõ‚(:¹ ÌÁæÔÄ¥ Ù network.recv‚ 1.879107e+06& disk.nvme0n1.out‚21974.65185546875 disk.rbd3.in‚0 cpu‚25.35831843165124 disk.rbd0.in‚0( disk.nvme0c0n1.out‚21974.65185546875# disk.dm-0.out‚21974.65185546875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚ 116.873046875 disk.dm-0.in‚ 176.6796875% disk./.usageGB‚1110.2403450012207 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.5390625 disk.sda1.in‚0* proc.memory.percent‚17.549419403076172 network.sent‚ 1.5314574e+07! disk.nvme0c0n1.in‚ 176.6796875 disk.nvme0n1.in‚ 176.6796875 disk.drbd1777.in‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0" memory_percent‚94.197678565979 disk.sda3.out‚ 199.12109375 disk.sda2.in‚0 disk.sda.out‚ 199.12109375( proc.memory.availableMB‚ 950.65234375 proc.cpu.threads‚17) disk./.usagePercent‚74.50916110672917 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0! proc.memory.rssMB‚ 2875.296875 disk.rbd2.in‚0 disk.dm-1.out‚ 199.12109375È¢­2ö- _wandbruntime‚5160  _runtime‚5160Ø ‚Æ÷À _runtime‚5160.615786427 _step‚539 & lr-AdamW/pg1‚9.348561767032284e-05 & lr-AdamW/pg2‚9.348561767032284e-05 trainer/global_step‚6599 $ _timestamp‚1.788453069061428e+09™Å5jËøÅ trainer/global_step‚6599 $ _timestamp‚1.788453069061428e+09 _runtime‚5160.615786427 _step‚539 & lr-AdamW/pg1‚9.348561767032284e-05 & lr-AdamW/pg2‚9.348561767032284e-05›-»/2ù- _wandbruntime‚5160  _runtime‚5160íCP5à'úÚO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55963260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 $ _timestamp‚1.788453069062302e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521910 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383548427954688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚6599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5795919723520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389073385254912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124563131 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95917088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5160.617040181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373500496362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418846356007424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088ž÷ßêå'ûßO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389073385254912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124563131 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95917088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5160.617040181 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373500496362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418846356007424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55963260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 $ _timestamp‚1.788453069062302e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521910 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383548427954688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚6599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5795919723520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0œ;¿ü- _wandbruntXɧKime‚5161  _runtime‚5161u±S…ê&ýäM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379060835099274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124561547 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5526451106816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31563196 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5796941749760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884530699760046e+09 _step‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383604762990592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844+LÖï&þéM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379061506576598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389131232100352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55971428 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220492649 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884530699760046e+09 _step‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383604762990592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5505451645578 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373555400493056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383604780993536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379060835099274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124561547 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5526451106816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5796776041984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87535077 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31563196 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5796941749760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88157420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421928916830720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386366079897088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788453105471155e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31786196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5197.025800373 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386366061894144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚427767951005696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5839034174976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5565961108992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386366061894144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚427767951005696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5839034174976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5565961108992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88157420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421928916830720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386366079897088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788453105471155e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31786196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5197.025800373 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 …á°2Œ- _wandbruntime‚5197  _runtime‚5197ñû×ë&åM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381847548444006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421990567568384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5197.691611155 _step‚545 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5567436912640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391989869848576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96631902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚381846876966682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96631904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 % _timestamp‚1.7884531061365292e+09W›¿ "ŽêM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391989869848576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391989869848576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96631902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6649 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚381846876966682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96631904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 % _timestamp‚1.7884531061365292e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376300607021056 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88170491 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381847548444006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546286984986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421990567568384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚391989195469824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5839896762368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5197.691611155 _step‚545 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5567436912640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56379828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceS²ôÙöÐtatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066¡ñjxú2- _wandbruntime‚5197  _runtime‚5197 z ^[Õ 1loss/FBt3/assignment_loss‚0.36272858349340303 1loss/SRqqt1/detection_loss‚1.8216468389419949 /loss/FRt1/assignment_loss‚2.712318564785851 0loss/FBt4/assignment_loss‚0.4982695006299764 0loss/SRqqt2/detection_loss‚1.332334638376471 epoch‚5 _step‚546 /loss/FBt1/detection_loss‚2.7412118673324586 2loss/SRqqt2/assignment_loss‚0.9444789760680475 /loss/FRt2/detection_loss‚1.6314360284987761 /loss/FRt4/detection_loss‚2.9106396064162254 /loss/FBt2/detection_loss‚0.7374696386115829 1loss/SRqqt4/detection_loss‚2.3836500576564243 % _timestamp‚1.7884531061383562e+09 /loss/FRt4/assignment_loss‚4.058089936152101 /loss/FRt3/detection_loss‚2.2483663358367405 2loss/SRqqt3/assignment_loss‚1.3215446089362275 2loss/SRqqt4/assignment_loss‚1.9009327730962209 0loss/FBt1/assignment_loss‚0.1873473978601396 1loss/FBt2/assignment_loss‚0.15810250040802282 /loss/FRt1/detection_loss‚2.5847020871109434 .loss/FBt4/detection_loss‚2.467965120077133 _runtime‚5197.692655754 .loss/FRt2/assignment_loss‚1.52813486055452 0loss/FRt3/assignment_loss‚2.4859363643022685 1loss/SRqqt3/detection_loss‚1.5578942663710693 /loss/FBt3/detection_loss‚2.6068961705480302 %loss/total_loss‚42.36225285343093 trainer/global_step‚6649 2loss/SRqqt1/assignment_loss‚1.1801561313663043àŽ,`‘Ú _step‚546 /loss/FBt1/detection_loss‚2.7412118673324586 2loss/SRqqt2/assignment_loss‚0.9444789760680475 /loss/FRt2/detection_loss‚1.6314360284987761 /loss/FRt4/detection_loss‚2.9106396064162254 /loss/FBt2/detection_loss‚0.7374696386115829 1loss/SRqqt4/detection_loss‚2.3836500576564243 % _timestamp‚1.7884531061383562e+09 /loss/FRt4/assignment_loss‚4.058089936152101 /loss/FRt3/detection_loss‚2.2483663358367405 2loss/SRqqt3/assignment_loss‚1.3215446089362275 2loss/SRqqt4/assignment_loss‚1.9009327730962209 0loss/FBt1/assignment_loss‚0.1873473978601396 1loss/FBt2/assignment_loss‚0.15810250040802282 /loss/FRt1/detection_loss‚2.5847020871109434 .loss/FBt4/detection_loss‚2.467965120077133 _runtime‚5197.692655754 .loss/FRt2/assignment_loss‚1.52813486055452 0loss/FRt3/assignment_loss‚2.4859363643022685 1loss/SRqqt3/detection_loss‚1.5578942663710693 /loss/FBt3/detection_loss‚2.6068961705480302 %loss/total_loss‚42.36225285343093 trainer/global_step‚6649 2loss/SRqqt1/assignment_loss‚1.1801561313663043 1loss/FBt3/assignment_loss‚0.36272858349340303 1loss/SRqqt1/detection_loss‚1.8216468389419949 /loss/FRt1/assignment_loss‚2.712318564785851 0loss/FBt4/assignment_loss‚0.4982695006299764 0loss/SRqqt2/detection_loss‚1.332334638376471 epoch‚5¢üeŽ1“’‚(:ˆ ùÁæÔþÞÅÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚103.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒúP€=“‚(:² ùÁæÔÞÞƒØ proc.cpu.threads‚17 network.recv‚ 1.892893e+06% disk./.usageGB‚1110.2401084899902 disk.rbd1.out‚ 2.6015625 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 176.6796875 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚94.19682025909424! proc.memory.rssMB‚ 2875.296875 disk.dm-1.out‚ 201.0234375 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875# disk.nvme0n1.out‚22170.34765625 disk.drbd1777.out‚0 cpu‚25.355678212871243 network.sent‚ 1.5434304e+07 disk.dm-0.out‚22170.34765625 disk.sda2.out‚0) disk./.usagePercent‚74.50914523426205 disk.rbd1.in‚0 disk.sda.out‚ 201.0234375 disk.sda3.out‚ 201.0234375! disk.nvme0c0n1.in‚ 176.6796875% disk.nvme0c0n1.out‚22170.34765625 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚118.62353515625 disk.nvme0n1.in‚ 176.6796875 disk.rbd3.in‚0( proc.memory.availableMB‚ 950.79296875* proc.memory.percent‚17.549419403076172Bd~“”‚(:ˆ ˆÂæÔœÐÉÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"sG•‚(:¼ ˆÂæÔ‘¿þØ disk.rbd1.out‚ 2.6337890625 disk.sda1.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚118.62353515625 disk.nvme0n1.in‚ 176.6796875& disk.nvme0n1.out‚22268.47314453125 network.recv‚ 1.899335e+06( disk./.usagePercent‚74.5091426741867 disk.dm-1.out‚ 201.43359375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 201.43359375% disk./.usageGB‚1110.2400703430176 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚22268.47314453125 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2875.296875 proc.cpu.threads‚17 network.sent‚ 1.554347e+07 disk.sda.out‚ 201.43359375 disk.sda1.in‚0* proc.memory.percent‚17.549419403076172# disk.dm-0.out‚22268.47314453125! disk.nvme0c0n1.in‚ 176.6796875 disk.dm-0.in‚ 176.6796875 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚94.19786930084229( proc.memory.availableMB‚ 950.62109375 cpu‚25.354746911319 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0Y(2–- _wandbruntime‚5233  _runtime‚5233ï*¢Ç—Á trainer/global_step‚6699 % _timestamp‚1.7884531417711086e+09 _runtime‚5233.325519902 _step‚547 & lr-AdamW/pg1‚9.082535521819917e-05 & lr-AdamW/pg2‚9.082535521819917e-05Ž9îÌ˜Æ trainer/global_step‚6699 % _timestamp‚1.7884531417711086e+09 _runtime‚5233.325519902 _step‚547 & lr-AdamW/pg1‚9.082535521819917e-05 & lr-AdamW/pg2‚9.082535521819917e-05£¼„’¾2™- _wandbruntime‚5233  _runtime‚5233³Dà'šÚO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223733923 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97318688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425011453713920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788453141771846e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378990892379136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012745 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5881926160896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223713883 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126415235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389183731839488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126395197 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440ànå'›ßO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5607603277312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5882093979136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚5233.326424076 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5607603277312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378990909418496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394790660751360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389183713836544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 _step‚548 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389183731839488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126395197 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223733923 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97318688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425011429773824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56780060 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425011453713920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788453141771846e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378990892379136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126415235 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88792807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012745 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5881926160896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384577869745682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223713883 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126415235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 trainer/global_step‚6699 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126413651 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_bkû㘰atch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚5234.010024019 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844uØfçð&žêM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚5234.010024019 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 trainer/global_step‚6699 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126413651 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126433715¥dY€T2Ÿ- _wandbruntime‚5234  _runtime‚5234²ÍžqY Ó 0loss/FRt3/assignment_loss‚2.9204638280519624 0loss/SRqqt2/detection_loss‚1.121772579421274 /loss/FBt2/detection_loss‚0.5943882073440095 /loss/FBt3/detection_loss‚2.7064594447612764 /loss/FRt4/assignment_loss‚4.156102132797241 %loss/total_loss‚44.42791156399758 2loss/SRqqt1/assignment_loss‚1.3718368692530527 2loss/SRqqt4/assignment_loss‚1.8896686828562193 /loss/FBt4/assignment_loss‚0.367401362338569 /loss/FBt1/detection_loss‚2.6771056056022644 /loss/FBt4/detection_loss‚2.8990724235773087 0loss/FRt2/assignment_loss‚1.5699170512973137 1loss/FBt1/assignment_loss‚0.38219899218529463 /loss/FRt2/detection_loss‚1.6682362661890264 /loss/FRt3/detection_loss‚2.9639033682462648 1loss/SRqqt3/detection_loss‚1.6065527861410718 epoch‚5 1loss/FBt3/assignment_loss‚0.23210759656503727 /loss/FRt4/detection_loss‚2.8743274807929993 0loss/SRqqt1/detection_loss‚1.727547236200836 trainer/global_step‚6699 /loss/FRt1/assignment_loss‚3.136852703664614 1loss/SRqqt3/assignment_loss‚1.593975548474294 /loss/FRt1/detection_loss‚2.4436729877539305 % _timestamp‚1.7884531424571648e+09 _runtime‚5234.011491866 0loss/SRqqt2/assignment_loss‚1.02803713779469 1loss/FBt2/assignment_loss‚0.19316418093702722 0loss/SRqqt4/detection_loss‚2.303147091752007 _step‚550*!*^¡Ø 0loss/FRt2/assignment_loss‚1.5699170512973137 1loss/FBt1/assignment_loss‚0.38219899218529463 /loss/FRt2/detection_loss‚1.6682362661890264 /loss/FRt3/detection_loss‚2.9639033682462648 1loss/SRqqt3/detection_loss‚1.6065527861410718 epoch‚5 1loss/FBt3/assignment_loss‚0.23210759656503727 /loss/FRt4/detection_loss‚2.8743274807929993 0loss/SRqqt1/detection_loss‚1.727547236200836 trainer/global_step‚6699 /loss/FRt1/assignment_loss‚3.136852703664614 1loss/SRqqt3/assignment_loss‚1.593975548474294 /loss/FRt1/detection_loss‚2.4436729877539305 % _timestamp‚1.7884531424571648e+09 _runtime‚5234.011491866 0loss/SRqqt2/assignment_loss‚1.02803713779469 1loss/FBt2/assignment_loss‚0.19316418093702722 0loss/SRqqt4/detection_loss‚2.303147091752007 _step‚550 0loss/FRt3/assignment_loss‚2.9204638280519624 0loss/SRqqt2/detection_loss‚1.121772579421274 /loss/FBt2/detection_loss‚0.5943882073440095 /loss/FBt3/detection_loss‚2.7064594447612764 /loss/FRt4/assignment_loss‚4.156102132797241 %loss/total_loss‚44.42791156399758 2loss/SRqqt1/assignment_loss‚1.3718368692530527 2loss/SRqqt4/assignment_loss‚1.8896686828562193 /loss/FBt4/assignment_loss‚0.367401362338569 /loss/FBt1/detection_loss‚2.6771056056022644 /loss/FBt4/detection_loss‚2.8990724235773087¦ß6z¢ž¢‚(:“ —ÂæÔïãÂÖ gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.953' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.590600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þuò?£‚(:´ —ÂæÔ‡¨ö×& disk.nvme0n1.out‚22268.81884765625! disk.rbd2.out‚118.70166015625# proc.memory.availableMB‚950.375 network.sent‚ 1.5548726e+07( disk./.usagePercent‚74.5091426741867 disk.dm-1.out‚ 202.078125 disk.rbd0.out‚0! proc.memory.rssMB‚ 2875.296875 cpu‚25.35388853474581 disk.rbd0.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚22268.81884765625 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.6337890625* proc.memory.percent‚17.549419403076172 proc.cpu.threads‚17( disk.nvme0c0n1.out‚22268.81884765625 disk.rbd3.out‚0 disk.sda1.in‚0% disk./.usageGB‚1110.2400703430176 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 202.078125 disk.rbd2.in‚0$ memory_percent‚94.19937133789062 network.recv‚ 1.902866e+06! disk.nvme0c0n1.in‚ 176.6796875 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 176.6796875 disk.dm-0.in‚ 176.6796875 disk.sda2.out‚0 disk.sda.out‚ 202.078125 disk.sda1.out‚0ÜÇVˤ‚(:’ ¦ÂæÔÒÐÈÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚97.692' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.538399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tg"ÁI¥‚(:¾ ¦ÂæÔã„ó×( disk./.usagePercent‚74.5091426741867 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.69677734375 disk.sda1.in‚0 cpu‚25.353024753999456 network.sent‚ 1.5657701e+07 disk.drbd1777.out‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 176.6796875" memory_percent‚94.199538230896 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 202.50390625 disk.rbd3.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚118.70166015625# disk.dm-0.out‚22366.67919921875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 202.50390625( disk.nvme0c0n1.out‚22366.67919921875* proc.memory.percent‚17.549419403076172 network.recv‚ 1.90873e+06% disk./.usageGB‚1110.2400703430176! disk.nvme0c0n1.in‚ 176.6796875 proc.cpu.threads‚17 disk.rbd2.in‚0& disk.nvme0n1.out‚22366.67919921875 disk.dm-0.in‚ 176.6796875 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 950.34765625! proc.memory.rssMB‚ 2875.296875 disk.sda3.out‚ 202.50390625˜K—'’¦‚(:‡ µÂæÔÿÂÜÕ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑAÿŒE§‚(:º µÂæÔ»ŽÝØ! disk.nvme0c0n1.in‚ 176.6796875 disk.dm-0.in‚ 176.6796875 proc.cpu.threads‚17! disk.rbd2.out‚120.33837890625 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 203.015625 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 203.015625( disk.nvme0c0n1.out‚22366.96435546875& disk.nvme0n1.out‚22366.96435546875 disk.drbd1777.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚22366.96435546875 disk.rbd1.out‚ 2.69677734375 disk.sda1.in‚0' proc.memory.availableMB‚ 950.8359375 network.sent‚ 1.5663373e+07% disk./.usageGB‚1110.2400703430176 disk.sda.in‚ 0.01171875 cpu‚25.352204613696504 disk.rbd0.in‚0 disk.nvme0n1.in‚ 176.6796875 disk.rbd1.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2875.296875 disk.sda.out‚ 203.015625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚94.19655799865723* proc.memory.percent‚17.549419403076172 network.recv‚ 1.912901e+06( disk./.usagePercent‚74.5091426741867)óï2¨- _wandbruntime‚5269  _runtime‚5269¸üÆ©À _step‚551 & lr-AdamW/pg1‚8.949549703167527e-05 & lr-AdamW/pg2‚8.949549703167527e-05 trainer/global_step‚6749 $ _timestamp‚1.788453177809408e+09 _runtime‚5269.363798695]ñON˪Š& lr-AdamW/pg1‚8.949549703167527e-05 & lr-AdamW/pg2‚8.949549703167527e-05 trainer/global_step‚6749 $ _timestamp‚1.788453177809408e+09 _runtime‚5269.363798695 _step‚551§]„vN2«- _wandbruntime‚5269  _runtime‚5269ž\^.x%¬ÛO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98019488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5627141228838 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434018898954240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127341287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5627795666594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397649298499584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚5269.365380867 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98019486 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884531778104477e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_po$/6 iol.allocated‚98019488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5924932297216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880èRÙvæ'­àO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5924932297216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98019488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57188460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5627141228838 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381736115946496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434018898954240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434019084421120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127341287 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5627795666594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397649298499584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647932720640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚5269.365380867 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98019486 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884531778104477e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98019488¨Nú|–2®- _wandbruntime‚5270  _runtime‚5270}}ý*ë&¯åM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32244181 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428155617394688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884531785007026e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5270.055627742 _step‚553 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397707145345024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚6749 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703¶:>ð&°êM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397706470966272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127359767 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57196630 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57196628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397707145345024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381791003037696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5649408524288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚6749 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32244181 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627957663802 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428155617394688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884531785007026e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5648752148480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5270.055627742 _step‚553 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128247301 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 # _timestamp‚1.78845321410449e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98720286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32466698 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98720288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98720286 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128247301 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57596860 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 # _timestamp‚1.78845321410449e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98720286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32466698 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98720288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98720286 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5306.333580779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394875370760192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32470807 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90076272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57605030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394875388763136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5668793003210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884532147784388e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384536209565696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inacúà kotive_split_bytes.all.allocated‚437207289677312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚557 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096‡Tþð&ÀêM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128285819 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5306.333580779 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394875370760192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128285819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32470807 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90076272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98734302 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57605030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394875388763136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5668793003210 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884532147784388e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384536209565696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437207289677312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400565783093248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚557 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 Fm2Í- _wandbruntime‚5341  _runtime‚5341åÌóà'ÎÚO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392935323870790 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6011095181312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387226511963136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99421086 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736Gæ8Àå ÏßO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397636687666688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403366573996032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99421086 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.Dìv5large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397636687666688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392935323870790 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434259040423424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99421086 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5730560694784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6011095181312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387226511963136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 J¨m½2Ð- _wandbruntime‚5342  _runtime‚5342!‡‘3ë&ÑåM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129191807 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228626909 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403424420841472 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392991044436314 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884532509191277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99435102 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397693022702592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58013428 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403423746462720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32699191 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5709628342618 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129191807 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228626909 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403424420841472 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32698744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392991044436314 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884532509191277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99435102 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397693022702592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58013428 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚561 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403423746462720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32699191 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚6849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5709628342618 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397693040705536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0±Jßô÷2Ó- _wandbruntime‚5342  _runtime‚5342Âê›s^ÔØ 2loss/SRqqt2/assignment_loss‚0.9457416459164747 1loss/SRqqt4/assignment_loss‚2.450347585303168 .loss/FRt3/detection_loss‚2.032070672382479 /loss/FBt1/detection_loss‚3.7550779581069946 /loss/FBt2/detection_loss‚0.5050128574959285 1loss/SRqqt3/assignment_loss‚1.486081571928386 /loss/FRt4/detection_loss‚3.1460982275009157 1loss/FBt2/assignment_loss‚0.15124870728991227 1loss/SRqqt1/detection_loss‚1.8643501529899926 /loss/FBt3/detection_loss‚2.0730156112800944 _runtime‚5342.475148219 0loss/FRt1/assignment_loss‚3.0720895188195363 0loss/FBt3/assignment_loss‚0.2851965093815862 0loss/FBt4/assignment_loss‚0.5441167615936138 1loss/SRqqt2/detection_loss‚0.9170803730102146 1loss/SRqqt3/detection_loss‚1.6683037543091281 epoch‚5 0loss/FRt2/assignment_loss‚1.4317565091320725 0loss/FRt4/assignment_loss‚3.4054634761810303 /loss/FRt2/detection_loss‚1.4254786162102808 0loss/FRt3/assignment_loss‚2.8668636977672577 %loss/total_loss‚43.42596020474391 _step‚562 1loss/FBt1/assignment_loss‚0.03505229732642571 1loss/SRqqt4/detection_loss‚2.5320014588294493 2loss/SRqqt1/assignment_loss‚1.2678936563002376 /loss/FRt1/detection_loss‚2.7343137289796555 /loss/FBt4/detection_loss‚2.8313048567090715 trainer/global_step‚6849 % _timestamp‚1.7884532509208272e+09­›sþcÕÝ /loss/FBt4/detection_loss‚2.8313048567090715 trainer/global_step‚6849 % _timestamp‚1.7884532509208272e+09 2loss/SRqqt2/assignment_loss‚0.9457416459164747 1loss/SRqqt4/assignment_loss‚2.450347585303168 .loss/FRt3/detection_loss‚2.032070672382479 /loss/FBt1/detection_loss‚3.7550779581069946 /loss/FBt2/detection_loss‚0.5050128574959285 1loss/SRqqt3/assignment_loss‚1.486081571928386 /loss/FRt4/detection_loss‚3.1460982275009157 1loss/FBt2/assignment_loss‚0.15124870728991227 1loss/SRqqt1/detection_loss‚1.8643501529899926 /loss/FBt3/detection_loss‚2.0730156112800944 _runtime‚5342.475148219 0loss/FRt1/assignment_loss‚3.0720895188195363 0loss/FBt3/assignment_loss‚0.2851965093815862 0loss/FBt4/assignment_loss‚0.5441167615936138 1loss/SRqqt2/detection_loss‚0.9170803730102146 1loss/SRqqt3/detection_loss‚1.6683037543091281 epoch‚5 0loss/FRt2/assignment_loss‚1.4317565091320725 0loss/FRt4/assignment_loss‚3.4054634761810303 /loss/FRt2/detection_loss‚1.4254786162102808 0loss/FRt3/assignment_loss‚2.8668636977672577 %loss/total_loss‚43.42596020474391 _step‚562 1loss/FBt1/assignment_loss‚0.03505229732642571 1loss/SRqqt4/detection_loss‚2.5320014588294493 2loss/SRqqt1/assignment_loss‚1.2678936563002376 /loss/FRt1/detection_loss‚2.7343137289796555²çi«’Ö‚(:‡ ÃæÔöãÕ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.555' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.511 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=©FJLׂ(:Á ÃæÔÕŽŒØ network.sent‚ 1.5903216e+07 disk.dm-1.out‚ 206.30078125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 176.68359375 disk.drbd1777.in‚0# disk.dm-0.out‚22765.80029296875 cpu‚25.347109486849593) disk./.usagePercent‚74.50961628812505 disk.rbd1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 176.68359375( disk.nvme0c0n1.out‚22765.80029296875 disk.drbd1777.out‚0$ memory_percent‚94.19877529144287( proc.memory.availableMB‚ 950.47265625 disk.sda2.in‚0! proc.memory.rssMB‚ 2875.296875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚122.2958984375* proc.memory.percent‚17.549419403076172$ disk./.usageGB‚1110.247127532959 disk.dm-0.in‚ 176.68359375 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 206.30078125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 206.30078125 network.recv‚ 1.941254e+06 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0& disk.nvme0n1.out‚22765.80029296875ñ%M“Ø‚(:ˆ žÃæÔ뿇Ø gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.511' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†~´ßBÙ‚(:® žÃæÔˆÐŒÚ disk.nvme0n1.out‚ 22863.90625 disk.rbd2.out‚122.3115234375 disk.drbd1777.in‚0 disk.dm-0.out‚ 22863.90625 disk.rbd1.out‚ 2.759765625 disk.sda3.out‚ 206.76171875 disk.rbd0.out‚0 network.sent‚ 1.6014409e+07 disk.dm-1.out‚ 206.76171875 disk.rbd3.in‚0 diUò÷sk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 206.76171875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.34626402113629 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0" disk.nvme0c0n1.out‚ 22863.90625) disk./.usagePercent‚74.50960451177848 disk.nvme0n1.in‚ 176.68359375 disk.sda3.in‚ 0.01171875* proc.memory.percent‚17.549419403076172 proc.cpu.threads‚17% disk./.usageGB‚1110.2469520568848 disk.dm-0.in‚ 176.68359375$ memory_percent‚94.20037269592285 network.recv‚ 1.947605e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 176.68359375' proc.memory.availableMB‚ 950.2109375! proc.memory.rssMB‚ 2875.296875G¸>2Ú- _wandbruntime‚5378  _runtime‚53788-j·ÇÛÁ _runtime‚5378.169102927 _step‚563 & lr-AdamW/pg1‚8.551033979760978e-05 & lr-AdamW/pg2‚8.551033979760978e-05 trainer/global_step‚6899 % _timestamp‚1.7884532866147144e+090y¡ÌÜÆ _runtime‚5378.169102927 _step‚563 & lr-AdamW/pg1‚8.551033979760978e-05 & lr-AdamW/pg2‚8.551033979760978e-05 trainer/global_step‚6899 % _timestamp‚1.7884532866147144e+09³¼ÄÒ$2Ý- _wandbruntime‚5378  _runtime‚5378°›:Áå'ÞßO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884532866154745e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5378.170106087 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91334919 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400454339609088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5771546500608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096…I:ê'ßäO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400454339609088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5771546500608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5771546500608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884532866154745e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5378.170106087 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91334919 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0´øR/(2à- _wandbruntime‚5378  _runtime‚5378I<™Yî&áèM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230273827 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400510692647936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390026639661056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5378.847413894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400510674644992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406283058589696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91348065 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406283058589696 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390026639661056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5378.847413894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400510674644992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406283058589696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91348065µŒ%¶b2ã- _wandbruntime‚5378  _runtime‚5378R´rÇYäÓ 1loss/FBt2/assignment_loss‚0.11584372075604257 /loss/FBt1/detection_loss‚2.4796850085258484 _runtime‚5378.848816297 _step‚566 1loss/SRqqt2/assignment_loss‚1.008715574766564 2loss/SRqqt3/assignment_loss‚1.3254308429800652 /loss/FBt2/detection_loss‚0.5268515924318973 0loss/FRt2/assignment_loss‚1.7939681055253012 1loss/FBt1/assignment_loss‚0.10007544575741381 1loss/FBt4/assignment_loss‚0.20116992926018107 0loss/SRqqt4/detection_loss‚2.484416219923231 trainer/global_step‚6899 /loss/FRt1/assignment_loss‚3.244634339544508 0loss/SRqqt4/assignment_loss‚2.14662294752068 0loss/FBt3/assignment_loss‚0.1490589641034603 /loss/FRt4/detection_loss‚3.1187509670853615 &loss/total_loss‚44.054170178241655 /loss/FRt3/detection_loss‚1.9922192772993674 1loss/SRqqt1/detection_loss‚1.8994721819460392 1loss/SRqqt2/detection_loss‚1.0998674879836685 /loss/FBt3/detection_loss‚2.9277445554733275 .loss/FBt4/detection_loss‚3.000802238782247 2loss/SRqqt1/assignment_loss‚1.2445753353461624 /loss/FRt3/assignment_loss‚2.977871047762724 1loss/SRqqt3/detection_loss‚1.7199198615821925 % _timestamp‚1.7884532872945495e+09 /loss/FRt4/assignment_loss‚4.620801478624344 .loss/FRt1/detection_loss‚2.316640168759558 .loss/FRt2/detection_loss‚1.559032886501467 epoch‚5Àæ\Š^åØ 0loss/FRt2/assignment_loss‚1.7939681055253012 1loss/FBt1/assignment_loss‚0.10007544575741381 1loss/FBt4/assignment_loss‚0.20116992926018107 0loss/SRqqt4/detection_loss‚2.484416219923231 trainer/global_step‚6899 /loss/FRt1/assignment_loss‚3.244634339544508 0loss/SRqqt4/assignment_loss‚2.14662294752068 0loss/FBt3/assignment_loss‚0.1490589641034603 /loss/FRt4/detection_loss‚3.1187509670853615 &loss/total_loss‚44.054170178241655 /loss/FRt3/detection_loss‚1.9922192772993674 1loss/SRqqt1/detection_loss‚1.8994721819460392 1loss/SRqqt2/detection_loss‚1.0998674879836685 /loss/FBt3/detection_loss‚2.9277445554733275 .loss/FBt4/detection_loss‚3.000802238782247 2loss/SRqqt1/assignment_loss‚1.2445753353461624 /loss/FRt3/assignment_loss‚2.977871047762724 1loss/SRqqt3/detection_loss‚1.7199198615821925 % _timestamp‚1.7884532872945495e+09 /loss/FRt4/assignment_loss‚4.620801478624344 .loss/FRt1/detection_loss‚2.316640168759558 .loss/FRt2/detection_loss‚1.559032886501467 epoch‚5 1loss/FBt2/assignment_loss‚0.11584372075604257 /loss/FBt1/detection_loss‚2.4796850085258484 _runtime‚5378.848816297 _step‚566 1loss/SRqqt2/assignment_loss‚1.008715574766564 2loss/SRqqt3/assignment_loss‚1.3254308429800652 /loss/FBt2/detection_loss‚0.5268515924318973¶0}œžæ‚(:“ ­ÃæÔæë±Õ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚157.999' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.599800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁŽwCç‚(:¸ ­ÃæÔ­à…Ø disk.rbd0.out‚0' disk.nvme0c0n1.out‚22864.2509765625 disk.drbd1777.in‚0 disk.dm-0.in‚ 176.68359375 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 176.68359375 disk.dm-1.in‚ 0.01171875$ memory_percent‚94.20382976531982* proc.memory.percent‚17.549419403076172 cpu‚25.34541643631199! disk.rbd2.out‚122.46044921875 disk.sda2.in‚0 disk.dm-1.out‚ 207.296875 disk.sda1.in‚0 disk.rbd1.out‚ 2.759765625! proc.memory.rssMB‚ 2875.296875 proc.cpu.threads‚17) disk./.usagePercent‚74.50962217629834$ disk./.usageGB‚1110.247215270996 disk.sda3.out‚ 207.296875" disk.nvme0c0n1.in‚ 176.68359375 disk.rbd1.in‚0 network.recv‚ 1.951964e+06 disk.rbd2.in‚0% disk.nvme0n1.out‚22864.2509765625" disk.dm-0.out‚22864.2509765625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 207.296875( proc.memory.availableMB‚ 949.64453125 network.sent‚ 1.6017881e+07 disk.sda3.in‚ 0.01171875äE2è‚(:’ ¼ÃæÔ¯ŽíÕ gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎœÎìGé‚(:¼ ¼ÃæÔ…¹¶Ø cpu‚25.34459595377264 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 208.1484375( proc.memory.availableMB‚ 949.63671875 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 176.68359375" disk.dm-0.out‚22962.1162109375$ memory_percent‚94.20387744903564* proc.memory.percent‚17.549419403076172% disk./.usageGB‚1110.2470016479492 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 208.1484375! disk.rbd2.out‚122.47607421875" disk.nvme0c0n1.in‚ 176.68359375' disk.nvme0c0n1.out‚22962.1162109375 disk.drbd1777.out‚0 network.recv‚ 1.958343e+06) disk./.usagePercent‚74.50960783987642 disk.sda3.out‚ 208.1484375 disk.nvme0n1.in‚ 176.68359375 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0% disk.nvme0n1.out‚22962.1162109375 disk.sda2.out‚0 network.sent‚ 1.6126956e+07 disk.rbd2.in‚0! proc.memory.rssMB‚ 2875.296875 disk.sda2.in‚0ÄœxV2ê- _wandbruntime‚5414  _runtime‚5414W´EÜÇëÁ _step‚567 & lr-AdamW/pg1‚8.418434927982784e-05 & lr-AdamW/pg2‚8.418434927982784e-05 trainer/global_step‚6949 % _timestamp‚1.7884533226700761e+09 _runtime‚5414.224475737¢k/YÌìÆ trainer/global_step‚6949 % _timestamp‚1.7884533226700761e+09 _runtime‚5414.224475737 _step‚567 & lr-AdamW/pg1‚8.418434927982784e-05 & lr-AdamW/pg2‚8.418434927982784e-05·?]L2í- _wandbruntime‚5414  _runtime‚5414Eäçå'îßO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100822686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409083849492480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91972007 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392716925019136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392716942058496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131025457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403271991551488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5811875943936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58822060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100822688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409084523857920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100822686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884533226708708e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791137024226 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _runtime‚5414.225553468 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91971555 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58822060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100822688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409084523857920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100822686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884533226708708e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791137024226 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _runtime‚5414.225553468 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91971555 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231880613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403328344590336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚5414.899459524 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5791299021434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409141021959168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 $ _timestamp‚1.788453323344214e+09 _step‚569 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813351747584 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403328344590336¼¯Fžó&òíM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231880613 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403328344590336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚5414.899459524 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440485764927488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100836702 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5791299021434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5812695371776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409141021959168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409141696337920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 $ _timestamp‚1.788453323344214e+09 _step‚569 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5813351747584 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403328344590336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58830230¹´xà2ó- _wandbruntime‚5414  _runtime‚5414 ø#Å]ô× 0loss/FRt1/assignment_loss‚2.9425066999486975 /loss/FRt2/assignment_loss‚2.101530825862518 1loss/FBt3/assignment_loss‚0.36349898540753534 _step‚570 2loss/FBt1/assignment_loss‚0.024982879999072338 -loss/FBt2/detection_loss‚0.50746679615501 trainer/global_step‚6949 1loss/SRqqt2/detection_loss‚1.0284332351293415 1loss/SRqqt4/detection_loss‚2.2885847012201945 1loss/SRqqt3/detection_loss‚1.9416514994894587 /loss/FBt1/detection_loss‚3.8156138261159263 .loss/FBt3/detection_loss‚2.128755710341714 %loss/total_loss‚44.71193354020816 /loss/FRt2/detection_loss‚1.9386173984608972 1loss/FBt4/assignment_loss‚0.44537543602928054 .loss/FBt4/detection_loss‚2.629394233226776 2loss/SRqqt1/assignment_loss‚1.5203777165710926 2loss/SRqqt2/assignment_loss‚1.1470638513892482 /loss/FRt4/detection_loss‚2.9698254199964658 _runtime‚5414.901063847 .loss/FRt1/detection_loss‚2.470992405873698 /loss/FRt4/assignment_loss‚3.605468911784036 2loss/SRqqt3/assignment_loss‚1.7648559627009601 2loss/SRqqt4/assignment_loss‚1.7062139424184959 1loss/FBt2/assignment_loss‚0.19679279410024625 /loss/FRt3/detection_loss‚2.0918151503592206 0loss/FRt3/assignment_loss‚3.2323313690068427 1loss/SRqqt1/detection_loss‚1.8497837886214257 epoch‚5 % _timestamp‚1.7884533233467717e+09‹”nbõÜ 2loss/FBt1/assignment_loss‚0.024982879999072338 -loss/FBt2/detection_loss‚0.50746679615501 trainer/global_step‚6949 1loss/SRqqt2/detection_loss‚1.0284332351293415 1loss/SRqqt4/detection_loss‚2.2885847012201945 1loss/SRqqt3/detection_loss‚1.9416514994894587 /loss/FBt1/detection_loss‚3.8156138261159263 .loss/FBt3/detection_loss‚2.128755710341714 %loss/total_loss‚44.71193354020816 /loss/FRt2/detection_loss‚1.9386173984608972 1loss/FBt4/assignment_loss‚0.44537543602928054 .loss/FBt4/detection_loss‚2.629394233226776 2loss/SRqqt1/assignment_loss‚1.5203777165710926 2loss/SRqqt2/assignment_loss‚1.1470638513892482 /loss/FRt4/detection_loss‚2.9698254199964658 _runtime‚5414.901063847 .loss/FRt1/detection_loss‚2.470992405873698 /loss/FRt4/assignment_loss‚3.605468911784036 2loss/SRqqt3/assignment_loss‚1.7648559627009601 2loss/SRqqt4/assignment_loss‚1.7062139424184959 1loss/FBt2/assignment_loss‚0.19679279410024625 /loss/FRt3/detection_loss‚2.0918151503592206 0loss/FRt3/assignment_loss‚3.2323313690068427 1loss/SRqqt1/detection_loss‚1.8497837886214257 epoch‚5 % _timestamp‚1.7884533233467717e+09 0loss/FRt1/assignment_loss‚2.9425066999486975 /loss/FRt2/assignment_loss‚2.101530825862518 1loss/FBt3/assignment_loss‚0.36349898540753534 _step‚570ºA9Æö‚(:’ ËÃæÔ„ÃÎÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.566000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ß2AB÷‚(:· ËÃæÔ¹«ŸØ disk.sda3.out‚ 208.703125 disk.sda.out‚ 208.703125 disk.rbd0.in‚0! disk.rbd2.out‚122.53857421875 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 208.703125' proc.memory.availableMB‚ 949.6640625! proc.memory.rssMB‚ 2875.296875 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1110.2470436096191 disk.rbd1.out‚ 2.759765625 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 176.68359375" disk.dm-0.out‚23059.5107421875$ memory_percent‚94.20371055603027 network.recv‚ 1.96191e+06( disk./.usagePercent‚74.5096106559593 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 176.68359375* proc.memory.percent‚17.549419403076172 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 cpu‚25.343785611121426 network.sent‚ 1.6132177e+07 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 176.68359375% disk.nvme0n1.out‚23059.5107421875 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0' disk.nvme0c0n1.out‚23059.5107421875 disk.drbd1777.out‚0q—(ò“ø‚(:ˆ ÚÃæÔ†ÖÎÕ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DºâÀIù‚(:¾ ÚÃæÔ¼ÿŠÙ disk.sda.out‚ 209.41015625 disk.rbd0.out‚0 disk.dm-0.in‚ 176.68359375 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875* proc.memory.percent‚17.549419403076172 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 176.68359375 disk.rbd1.in‚0$ memory_percent‚94.20256614685059$ disk./.usageGB‚1110.246295928955 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2875.296875 cpu‚25.343015580590457 network.sent‚ 1.6241177e+07) disk./.usagePercent‚74.50956047848258! disk.rbd2.out‚124.06201171875" disk.dm-0.out‚23060.0283203125 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 2.759765625 disk.drbd1777.out‚0 network.recv‚ 1.968451e+06 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 176.68359375 disk.sda2.in‚0 disk.sda3.out‚ 209.41015625' proc.memory.availableMB‚ 949.8515625 proc.cpu.threads‚17 disk.dm-1.out‚ 209.41015625 disk.sda.in‚ 0.01171875 disk.sda1.out‚0' disk.nvme0c0n1.out‚23060.0283203125% disk.nvme0n1.out‚23060.0283203125 disk.sda2.out‚0 disk.rbd3.in‚0F…vûžú‚(:“ éÃæÔâõŠÖ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.559399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)NûKû‚(:À éÃæÔÊØÐØ cpu‚25.342170774712294 network.sent‚ 1.6246774e+07) disk./.usagePercent‚74.50955535833191 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚ 176.68359375 network.recv‚ 1.972543e+06$ memory_percent‚94.20406818389893% disk.nvme0n1.out‚23157.5400390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda3.out‚ 210.00390625! disk.rbd2.out‚124.13232421875" disk.nvme0c0n1.in‚ 176.68359375 disk.nvme0n1.in‚ 176.68359375! proc.memory.rssMB‚ 2875.296875 disk.dm-1.out‚ 210.00390625 disk.sda.out‚ 210.00390625 disk.sda1.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.549419403076172 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1110.2462196350098 disk.sda2.out‚0 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0' disk.nvme0c0n1.out‚23157.5400390625" disk.dm-0.out‚23157.5400390625( proc.memory.availableMB‚ 949.60546875®@àd2ü- _wandbruntime‚5450  _runtime‚5450ˆšÆýÀ _step‚571 & lr-AdamW/pg1‚8.286001716330257e-05 & lr-AdamW/pg2‚8.286001716330257e-05 trainer/global_step‚6999 $ _timestamp‚1.788453359065652e+09 _runtime‚5450.620018813‚‹š¤ËþÅ _runtime‚5450.620018813 _step‚571 & lr-AdamW/pg1‚8.286001716330257e-05 & lr-AdamW/pg2‚8.286001716330257e-05 trainer/global_step‚6999 $ _timestamp‚1.788453359065652e+09»Ú7y.2ÿ- _wandbruntime‚5450  _runtime‚5450å;tXù€ßO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131951509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 O@DeviceStatsMonitor.on_train_batch_start/a°}Iøì llocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411943161606144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59230460 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884533590664012e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101523488 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406089643493888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831317925878 trainer/global_step‚6999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59230462 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5450.620939758 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33377739 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443506675013120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101523486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131971547 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430Ë»¯eê'äO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411943161606144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59230460 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884533590664012e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101523488 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406089643493888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831317925878 trainer/global_step‚6999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59230462 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5450.620939758 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33377739 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443506675013120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101523486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131971547 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131951509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406089625490944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831972363634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430¼¥=92‚- _wandbruntime‚5451  _runtime‚5451Y>ºí&ƒçM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406145978529792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5451.309602865 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443568301810688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5853681177600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101537504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832134360842 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6141870305792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088µ¹pè„ìM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832788798806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406145996532736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5451.309602865 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5854337553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443568301810688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚411999659707392 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5853681177600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101537504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412000334086144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401349170038538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832134360842 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6141870305792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406145978529792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446589211896320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132877561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235121887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚7049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5894503918080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408907295436288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102224286 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404080162817538 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414801799354368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5872807703042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5893847555584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5893847555584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5894503918080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408907295436288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102224286 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404080162817538 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102224286 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414801799354368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5872807703042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5893847555584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 142652416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5893847555584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59638862 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398207338075136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446589211896320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132877561 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93243024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680ÀOÎ+Ã’- _wandbruntime‚5487ûõ…7  _runtime‚5487>„/:ï&“éM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102238302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102238304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚7049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5895323359232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398262242205696 _runtime‚5487.649131912 _step‚577 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376õøô&”îM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102238302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102238304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚7049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5895323359232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398262242205696 _runtime‚5487.649131912 _step‚577 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132916079 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132896015 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÁÝèÓð2•- _wandbruntime‚5487  _runtime‚5487OÌGW–Ñ 2loss/SRqqt4/assignment_loss‚1.7973343544992908 2loss/SRqqt3/assignment_loss‚1.3680510778538884 1loss/SRqqt1/detection_loss‚1.7698906410175066 1loss/SRqqt2/detection_loss‚0.9826156027735241 /loss/FBt2/detection_loss‚0.5051981534548782 _runtime‚ 5487.65045252 0loss/FRt1/assignment_loss‚2.8556656814538517 2loss/SRqqt2/assignment_loss‚0.7948866021188516 1loss/FBt1/assignment_loss‚0.36997657505774195 -loss/FBt3/detection_loss‚2.29364253282547 .loss/FBt4/detection_loss‚2.349814079024575 -loss/FRt2/detection_loss‚1.71055216166903 _step‚578 0loss/FRt2/assignment_loss‚1.5556679870568069 /loss/FRt4/assignment_loss‚3.919550393566941 1loss/FBt3/assignment_loss‚0.30079215120385927 /loss/FRt1/detection_loss‚2.2152979995768804 .loss/FBt1/detection_loss‚2.692112071947618 trainer/global_step‚7049 $ _timestamp‚1.788453396096091e+09 2loss/SRqqt1/assignment_loss‚1.1645984233667452 0loss/FBt2/assignment_loss‚0.1549129291128521 1loss/FBt4/assignment_loss‚0.32106269080064853 /loss/SRqqt3/detection_loss‚1.73197474360466 1loss/SRqqt4/detection_loss‚2.1167734871650565 &loss/total_loss‚41.210927118293256 /loss/FRt3/detection_loss‚2.1036283587867564 /loss/FRt3/assignment_loss‚3.319739131494002 /loss/FRt4/detection_loss‚2.8171892888618237 epoch‚5Âkøý\—Ö 0loss/FRt1/assignment_loss‚2.8556656814538517 2loss/SRqqt2/assignment_loss‚0.7948866021188516 1loss/FBt1/assignment_loss‚0.36997657505774195 -loss/FBt3/detection_loss‚2.29364253282547 .loss/FBt4/detection_loss‚2.349814079024575 -loss/FRt2/detection_loss‚1.71055216166903 _step‚578 0loss/FRt2/assignment_loss‚1.5556679870568069 /loss/FRt4/assignment_loss‚3.919550393566941 1loss/FBt3/assignment_loss‚0.30079215120385927 /loss/FRt1/detection_loss‚2.2152979995768804 .loss/FBt1/detection_loss‚2.692112071947618 trainer/global_step‚7049 $ _timestamp‚1.788453396096091e+09 2loss/SRqqt1/assignment_loss‚1.1645984233667452 0loss/FBt2/assignment_loss‚0.1549129291128521 1loss/FBt4/assignment_loss‚0.32106269080064853 /loss/SRqqt3/detection_loss‚1.73197474360466 1loss/SRqqt4/detection_loss‚2.1167734871650565 &loss/total_loss‚41.210927118293256 /loss/FRt3/detection_loss‚2.1036283587867564 /loss/FRt3/assignment_loss‚3.319739131494002 /loss/FRt4/detection_loss‚2.8171892888618237 epoch‚5 2loss/SRqqt4/assignment_loss‚1.7973343544992908 2loss/SRqqt3/assignment_loss‚1.3680510778538884 1loss/SRqqt1/detection_loss‚1.7698906410175066 1loss/SRqqt2/detection_loss‚0.9826156027735241 /loss/FBt2/detection_loss‚0.5051981534548782 _runtime‚ 5487.65045252ÂD:ª’˜‚(:‡ –ÄæÔðãóÖ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚98.054' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p“L™‚(:Á –ÄæÔ™¾ñÙ disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚23353.71142578125( proc.memory.availableMB‚ 949.64453125 disk.dm-0.in‚ 176.68359375# disk.dm-0.out‚23353.71142578125 disk.sda.out‚ 212.07421875* proc.memory.percent‚17.549419403076172 network.recv‚ 1.98714e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 2.759765625 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 176.68359375! proc.memory.rssMB‚ 2875.296875 disk.rbd2.in‚0 network.sent‚ 1.6380585e+07( disk./.usagePercent‚74.5095538222867 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.339774971018368& disk.nvme0n1.out‚23353.71142578125 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1110.2461967468262 disk.sda1.in‚0$ memory_percent‚94.20382976531982 proc.cpu.threads‚17! disk.rbd2.out‚125.59814453125 disk.drbd1777.in‚0 disk.sda3.out‚ 212.07421875 disk.nvme0n1.in‚ 176.68359375 disk.dm-1.out‚ 212.07421875 disk.rbd3.out‚0 disk.sda1.out‚0~Ußš‚(:… ¥ÄæÔã™àÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.25' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.25 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*L|¡<›‚(:± ¥ÄæÔÀÇØ network.recv‚ 1.993691e+06 disk.rbd0.in‚0! proc.memory.rssMB‚ 2875.296875 disk.sda3.out‚ 212.5078125 disk.sda2.in‚0 disk.dm-1.out‚ 212.5078125 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 949.75390625 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.33900179514102 proc.cpu.threads‚17( disk./.usagePercent‚74.5095538222867% disk./.usageGB‚1110.2461967468262 disk.sda2.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda.out‚ 212.5078125$ disk.nvme0c0n1.out‚ 23354.0390625 disk.rbd0.out‚0! disk.rbd2.out‚125.66064453125" disk.nvme0n1.out‚ 23354.0390625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.549419403076172 network.sent‚ 1.647592e+07" disk.nvme0c0n1.in‚ 176.68359375 disk.dm-0.in‚ 176.68359375 disk.dm-0.out‚ 23354.0390625 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875$ memory_percent‚94.20316219329834 disk.nvme0n1.in‚ 176.68359375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0\ö”Ëžœ‚(:“ ´ÄæÔüø½Õ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.080400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ F‚(:» ´ÄæÔý¾Ý× disk.sda3.out‚ 212.9765625 disk.rbd1.out‚ 2.759765625" disk.nvme0c0n1.in‚ 176.68359375 disk.dm-1.in‚ 0.01171875' disk.nvme0c0n1.out‚23452.1748046875$ memory_percent‚94.20168399810791! proc.memory.rssMB‚ 2875.296875 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 network.sent‚ 1.6482264e+07 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 212.9765625" disk.dm-0.out‚23452.1748046875 disk.sda.out‚ 212.9765625 cpu‚25.338227957121152 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.998079e+06! disk.rbd2.out‚125.66455078125% disk.nvme0n1.out‚23452.1748046875* proc.memory.percent‚17.549419403076172 disk.sda2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚74.50954972616614# disk./.usageGB‚1110.24613571167 disk.rbd3.in‚0 disk.nvme0n1.in‚ 176.68359375 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 176.68359375( proc.memory.availableMB‚ 949.99609375 disk.sda3.in‚ 0.0117187532ž- _wandbruntime‚5523  _runtime‚5523?©2üÇŸÁ _runtime‚5523.500355975 _step‚579 & lr-AdamW/pg1‚8.021743168621048e-05 & lr-AdamW/pg2‚8.021743168621048e-05 trainer/global_step‚7099 % _timestamp‚1.7884534319459279e+09¹–6]Ì Æ _step‚579 & lr-AdamW/pg1‚8.021743168621048e-05 & lr-AdamW/pg2‚8.021743168621048e-05 trainer/global_step‚7099 % _timestamp‚1.7884534319459279e+09 _runtime‚5523.500355975âý"¶2¡- _wandbruntime‚5523  _runtime‚5523òÈ…)¢ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚7099 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884534319467554e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411724929375744 WJDeviceStatsMonitor.on_train_ñ,U¼batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133803613 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚580 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5934833361408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5523.501404371 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236728699 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417660437102592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6227100449280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414óJê'£äO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚7099 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411724947378688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884534319467554e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411724929375744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5935489723904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133803613 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚580 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5934833361408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5523.501404371 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236728699 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417660437102592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚449671724839424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93879598 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6227100449280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946ÄW©2¤- _wandbruntime‚5524  _runtime‚5524ˆëï&¥éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133822067 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5914459477622 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455961342809088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133842131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚455961526165504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133822067 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 FyǯÁ & lr-AdamW/pg1‚7.889972903649628e-05 & lr-AdamW/pg2‚7.889972903649628e-05 trainer/global_step‚7149 % _timestamp‚1.7884534683546894e+09 _runtime‚5559.909060817 _step‚583‡ÒÐFÌ°Æ % _timestamp‚1.7884534683546894e+09 _runtime‚5559.909060817 _step‚583 & lr-AdamW/pg1‚7.889972903649628e-05 & lr-AdamW/pg2‚7.889972903649628e-05 trainer/global_step‚7149Ç`xû¿2±- _wandbruntime‚5559  _runtime‚5559kÖå'²ßO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270326866944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420518400485376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 _step‚584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94516376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409652246552354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5954478381858 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5559.910168014 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103625888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414542581318144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884534683554752e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414542599321088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134729665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÏÎœ¹ê'³äO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414542599321088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134729665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103625886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6270326866944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420518400485376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 _step‚584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94516376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409652246552354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5954478381858 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5559.910168014 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103625888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414542581318144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884534683554752e+09ÈŒe±2´- _wandbruntime‚5560  _runtime‚5560Ib¸êû µéM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60463828 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403752672301056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 FDeviceStatsMonitor.on_train_bIš“ªôatch_end/inactive_split.all.freed‚94529101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚585 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420576247330816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414598952359936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884534692704105e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚5560.825567396 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34065273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409707967118086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459087120513536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103639904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403752655261696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60463830 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0­0KAô&¶îM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459087120513536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6271014211072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5977294970880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103639904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403752655261696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60463830 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60463828 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚452815936400384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134768183 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403752672301056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚585 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94529564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420576247330816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414598952359936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884534692704105e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚5560.825567396 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5977294970880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34065273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409707967118086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Éøšû2·- _wandbruntime‚5560  _runtime‚5560‹ö+B[¸Õ epoch‚5 $ _timestamp‚1.788453469272342e+09 0loss/FBt4/assignment_loss‚0.3686851190983361 /loss/FRt2/detection_loss‚1.3955772118642926 0loss/FRt1/assignment_loss‚2.0393508034093037 0loss/FRt3/assignment_loss‚3.0078984726042974 0loss/FRt4/assignment_loss‚3.9970115780830384 1loss/SRqqt1/assignment_loss‚1.410332323166362 2loss/SRqqt4/assignment_loss‚1.4186174883957832 1loss/FBt1/assignment_loss‚0.06959629608900286 .loss/FBt1/detection_loss‚2.363492250442505 /loss/FBt3/detection_loss‚2.4103287637233732 /loss/FBt4/detection_loss‚2.7087090611457825 %loss/total_loss‚41.25369602649289 0loss/FBt3/assignment_loss‚0.9791424758732319 /loss/FRt1/detection_loss‚2.1700374292475835 1loss/SRqqt3/detection_loss‚1.7124857360290155 /loss/FBt2/detection_loss‚0.6617584600753617 2loss/SRqqt3/assignment_loss‚0.9136049500494091 0loss/SRqqt2/detection_loss‚1.054710261733599 1loss/SRqqt4/detection_loss‚2.8673812939274694 0loss/FRt2/assignment_loss‚1.3797830194234848 /loss/FRt4/detection_loss‚3.2008883535861967 _runtime‚5560.826734375 1loss/SRqqt2/assignment_loss‚1.030246465804685 1loss/FBt2/assignment_loss‚0.16778387089791214 trainer/global_step‚7149 .loss/FRt3/detection_loss‚2.313560988931429 1loss/SRqqt1/detection_loss‚1.6127133528914368 _step‚586êòFÈ`¹Ú 0loss/FRt2/assignment_loss‚1.3797830194234848 /loss/FRt4/detection_loss‚3.2008883535861967 _runtime‚5560.826734375 1loss/SRqqt2/assignment_loss‚1.030246465804685 1loss/FBt2/assignment_loss‚0.16778387089791214 trainer/global_step‚7149 .loss/FRt3/detection_loss‚2.313560988931429 1loss/SRqqt1/detection_loss‚1.6127133528914368 _step‚586 epoch‚5 $ _timestamp‚1.788453469272342e+09 0loss/FBt4/assignment_loss‚0.3686851190983361 /loss/FRt2/detection_loss‚1.3955772118642926 0loss/FRt1/assignment_loss‚2.0393508034093037 0loss/FRt3/assignment_loss‚3.0078984726042974 0loss/FRt4/assignment_loss‚3.9970115780830384 1loss/SRqqt1/assignment_loss‚1.410332323166362 2loss/SRqqt4/assignment_loss‚1.4186174883957832 1loss/FBt1/assignment_loss‚0.06959629608900286 .loss/FBt1/detection_loss‚2.363492250442505 /loss/FBt3/detection_loss‚2.4103287637233732 /loss/FBt4/detection_loss‚2.7087090611457825 %loss/total_loss‚41.25369602649289 0loss/FBt3/assignment_loss‚0.9791424758732319 /loss/FRt1/detection_loss‚2.1700374292475835 1loss/SRqqt3/detection_loss‚1.7124857360290155 /loss/FBt2/detection_loss‚0.6617584600753617 2loss/SRqqt3/assignment_loss‚0.9136049500494091 0loss/SRqqt2/detection_loss‚1.054710261733599 1loss/SRqqt4/detection_loss‚2.8673812939274694ÊÌQ³º‚(:’ áÄæÔÞÓŸÖ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.77' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.554000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áìwKP»‚(:Å áÄæÔµñ¸Ù disk.dm-1.out‚ 214.77734375) disk./.usagePercent‚74.50958531121341 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚23647.52294921875 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0! disk.rbd2.out‚125.86865234375* proc.memory.percent‚17.549443244934082 proc.cpu.threads‚17% disk./.usageGB‚1110.2466659545898 disk.rbd3.in‚0( disk.nvme0c0n1.out‚23647.52294921875# disk.dm-0.out‚23647.52294921875( proc.memory.availableMB‚ 949.64453125 network.sent‚ 1.6602106e+07 disk.rbd1.in‚0 disk.sda.out‚ 214.77734375 disk.sda1.out‚0 cpu‚25.335936480531608 disk.sda3.out‚ 214.77734375" disk.nvme0c0n1.in‚ 176.68359375# proc.memory.rssMB‚ 2875.30078125 disk.nvme0n1.in‚ 176.68359375 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚94.20382976531982 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 176.68359375 network.recv‚ 2.012249e+06 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0 2`>ž¼‚(:“ ðÄæÔ×ûÓÕ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.522' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.704399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ûÄCB½‚(:· ðÄæÔЋëØ disk.rbd0.in‚0 disk.dm-0.out‚23648.09765625 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.30078125 disk.drbd1777.out‚0 disk.dm-0.in‚ 176.68359375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme0c0n1.out‚23648.09765625 disk.nvme0n1.in‚ 176.68359375* proc.memory.percent‚17.549443244934082 network.recv‚ 2.018622e+06" disk.nvme0c0n1.in‚ 176.68359375 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.out‚ 215.2109375 network.sent‚ 1.6711311e+07) disk./.usagePercent‚74.50958531121341! disk.rbd2.out‚125.86865234375 disk.rbd1.in‚0 disk.sda1.in‚0" memory_percent‚94.203782081604% disk./.usageGB‚1110.2466659545898 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 949.65234375 disk.drbd1777.in‚0 disk.dm-1.out‚ 215.2109375 disk.rbd1.out‚ 2.759765625 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 215.2109375# disk.nvme0n1.out‚23648.09765625 cpu‚25.335177197098982"Q˜“¾‚(:ˆ ÿÄæÔ±¡íÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ÙÎüD¿‚(:¹ ÿÄæÔåÖ‰Ø disk.rbd0.out‚0# proc.memory.rssMB‚ 2875.30078125 network.sent‚ 1.6716587e+07 network.recv‚ 2.022447e+06 disk.sda3.out‚ 215.83203125 disk.rbd2.in‚0$ disk.nvme0c0n1.out‚ 23745.5703125" disk.nvme0n1.out‚ 23745.5703125 disk.dm-1.out‚ 215.83203125$ memory_percent‚94.20526027679443 disk.sda.out‚ 215.83203125* proc.memory.percent‚17.549443244934082 disk.rbd3.in‚0 disk.dm-0.in‚ 176.68359375 disk.dm-0.out‚ 23745.5703125 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.50958710326616% disk./.usageGB‚1110.2466926574707 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.dm-1.in‚ 0.01171875( proc.memory.availableMB‚ 949.41015625 cpu‚25.334423173248826 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚125.88818359375 disk.nvme0n1.in‚ 176.68359375 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 176.68359375óaà:2À- _wandbruntime‚5596  _runtime‚55963‚ÆÁÀ _step‚587 & lr-AdamW/pg1‚7.758478609497668e-05 & lr-AdamW/pg2‚7.758478609497668e-05 trainer/global_step‚7199 $ _timestamp‚1.788453504899847e+09 _runtime‚5596.454196694ÓtÿËÂÅ & lr-AdamW/pg1‚7.758478609497668e-05 & lr-AdamW/pg2‚7.758478609497668e-05 trainer/global_step‚7199 $ _timestamp‚1.788453504899847e+09 _runtime‚5596.454196694 _step‚587ˇyp2Ã- _wandbruntime‚5596  _runtime‚5596÷iC£"ÄÞO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423377038233600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417360251263488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417360233260544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313414849536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884535049005558e+09 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95153449 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135655717 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423377712599040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60864062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 5596.45514996 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚7199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455836798605824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406442957659136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104326686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60864060 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104326688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.smalW™ÝnÂl_pool.allocated‚ 6017461335552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688h8¯ñé'ÅãO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135655717 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423377712599040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60864062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 5596.45514996 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6313238642688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚7199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455836798605824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406442957659136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104326686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60864060 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104326688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6017461335552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423377038233600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417360251263488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417360251263488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417360233260544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313414849536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884535049005558e+09 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95153449 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928ÌË„²½2Æ- _wandbruntime‚5597  _runtime‚5597¬Ì—ï&ÇéM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6314119564800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240014873 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104340704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34293432 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406497878829056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚589 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚5597.125875144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135694235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60872230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423434210700288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018280776704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884535055708852e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412494008985494 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844çÉž«ô&ÈîM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412494008985494 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104340702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6314119564800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240014873 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104340704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34293432 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406497878829056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚589 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚5597.125875144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135694235 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60872230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423434210700288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018280776704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884535055708852e+09Í.(?2É- _wandbruntime‚5597  _runtime‚5597üÎX"ÝÊÑ 0loss/FRt4/assignment_loss‚3.8374295149530684 2loss/SRqqt1/assignment_loss‚1.8196338886608843 2loss/SRqqt4/assignment_loss‚2.0854958721569608 1loss/SRqqt3/detection_loss‚1.7127347796851276 .loss/Fý1sÇzBt1/detection_loss‚2.118198908865452 0loss/FBt1/assignment_loss‚0.3606305650318973 epoch‚5 trainer/global_step‚7199 % _timestamp‚1.7884535055725892e+09 _step‚590 1loss/SRqqt4/detection_loss‚2.4231220462492535 /loss/FBt2/detection_loss‚0.4951444271990147 0loss/FRt1/assignment_loss‚2.5908190721557256 1loss/FBt3/assignment_loss‚0.41188996104109304 0loss/FBt4/assignment_loss‚0.1907033009119905 %loss/total_loss‚43.19825925509539 /loss/FRt2/assignment_loss‚1.586709862947464 2loss/SRqqt2/assignment_loss‚0.9090446272919721 2loss/SRqqt3/assignment_loss‚1.6135245562359994 .loss/FRt1/detection_loss‚2.375827990117527 .loss/FRt2/detection_loss‚2.006844028457999 .loss/FBt4/detection_loss‚2.855037122964859 _runtime‚ 5597.12687073 .loss/FRt3/assignment_loss‚3.43588095108668 1loss/FBt2/assignment_loss‚0.18141920931111652 .loss/FRt3/detection_loss‚2.087869057556379 .loss/FRt4/detection_loss‚3.020772310239928 1loss/SRqqt1/detection_loss‚1.6345941138560656 1loss/SRqqt2/detection_loss‚1.1837587735670454 /loss/FBt3/detection_loss‚2.2611743145518832L÷m|\ËÖ 0loss/FBt1/assignment_loss‚0.3606305650318973 epoch‚5 trainer/global_step‚7199 % _timestamp‚1.7884535055725892e+09 _step‚590 1loss/SRqqt4/detection_loss‚2.4231220462492535 /loss/FBt2/detection_loss‚0.4951444271990147 0loss/FRt1/assignment_loss‚2.5908190721557256 1loss/FBt3/assignment_loss‚0.41188996104109304 0loss/FBt4/assignment_loss‚0.1907033009119905 %loss/total_loss‚43.19825925509539 /loss/FRt2/assignment_loss‚1.586709862947464 2loss/SRqqt2/assignment_loss‚0.9090446272919721 2loss/SRqqt3/assignment_loss‚1.6135245562359994 .loss/FRt1/detection_loss‚2.375827990117527 .loss/FRt2/detection_loss‚2.006844028457999 .loss/FBt4/detection_loss‚2.855037122964859 _runtime‚ 5597.12687073 .loss/FRt3/assignment_loss‚3.43588095108668 1loss/FBt2/assignment_loss‚0.18141920931111652 .loss/FRt3/detection_loss‚2.087869057556379 .loss/FRt4/detection_loss‚3.020772310239928 1loss/SRqqt1/detection_loss‚1.6345941138560656 1loss/SRqqt2/detection_loss‚1.1837587735670454 /loss/FBt3/detection_loss‚2.2611743145518832 0loss/FRt4/assignment_loss‚3.8374295149530684 2loss/SRqqt1/assignment_loss‚1.8196338886608843 2loss/SRqqt4/assignment_loss‚2.0854958721569608 1loss/SRqqt3/detection_loss‚1.7127347796851276 .loss/FBt1/detection_loss‚2.118198908865452Îòî]›’Ì‚(:‡ ŽÅæÔ¶»ÆÕ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦(foPÍ‚(:Å ŽÅæÔú™˜Ù( proc.memory.availableMB‚ 949.58984375 network.sent‚ 1.6722793e+07 network.recv‚ 2.026861e+06! disk.rbd2.out‚125.88818359375( disk.nvme0c0n1.out‚23843.95556640625 disk.dm-0.in‚ 176.68359375# disk.dm-0.out‚23843.95556640625 disk.dm-1.out‚ 216.26171875 disk.sda.in‚ 0.01171875% disk./.usageGB‚1110.2467193603516 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.50958889531888 disk.rbd2.in‚0 disk.sda2.in‚0$ memory_percent‚94.20416355133057 cpu‚25.333655844106815& disk.nvme0n1.out‚23843.95556640625 disk.drbd1777.out‚0* proc.memory.percent‚17.549443244934082 disk.rbd0.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2875.30078125 disk.sda.out‚ 216.26171875 disk.sda3.out‚ 216.26171875" disk.nvme0c0n1.in‚ 176.68359375 disk.nvme0n1.in‚ 176.68359375 disk.sda2.out‚0 disk.rbd1.out‚ 2.759765625 Çê‘΂(:† ÅæÔïÀªÕ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hý0NÏ‚(:à ÅæÔâÈÙ( disk.nvme0c0n1.out‚23844.19775390625 network.sent‚ 1.6831056e+07 network.recv‚ 2.032699e+06 disk.dm-1.out‚ 216.95703125 disk.rbd1.out‚ 2.759765625 disk.rbd0.out‚0! disk.rbd2.out‚125.95068359375& disk.nvme0n1.out‚23844.19775390625% disk./.usageGB‚1110.2467651367188 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2875.30078125* proc.memory.percent‚17.549443244934082) disk./.usagePercent‚74.50959196740929 disk.nvme0n1.in‚ 176.68359375$ memory_percent‚94.20266151428223 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 176.68359375 cpu‚25.33290453985961 proc.cpu.threads‚17 disk.sda3.out‚ 216.95703125" disk.nvme0c0n1.in‚ 176.68359375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚23844.19775390625 disk.sda2.out‚0' proc.memory.availableMB‚ 949.8359375 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.out‚ 216.95703125Va8ë2Ð- _wandbruntime‚5632  _runtime‚5632ömáÆÑÀ & lr-AdamW/pg1‚7.627287689378982e-05 & lr-AdamW/pg2‚7.627287689378982e-05 trainer/global_step‚7249 $ _timestamp‚1.788453541239904e+09 _runtime‚5632.794428575 _step‚591å„´ÚËÒÅ & lr-AdamW/pg2‚7.627287689378982e-05 trainer/global_step‚7249 $ _timestamp‚1.788453541239904e+09 _runtime‚5632.794428575 _step‚591 & lr-AdamW/pg1‚7.627287689378982e-05Ï"¡¡2Ó- _wandbruntime‚5632  _runtime‚5632`'/Šå'ÔßO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5632.795418651 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95788676 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34516654 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6356329098240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356486430720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420177903205888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889÷Ð7øê'ÕäO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚420177903205888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95789116 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105027488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105027486 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136581769 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356486430720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420177903205888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5632.795418651 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95788676 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34516654 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6356329098240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚420234256244736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚7249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batc®×„Þh_end/inactive_split_bytes.all.current‚ 193842176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6036965495846 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884535419218922e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34520644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056þYã—ô&ØîM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6357352033792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚420234238241792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6036965495846 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884535419218922e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34520644 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458980986226688 _runtime‚5633.477142559 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136620287 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚420234256244736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚7249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 PDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34761524 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6378745542144 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚411192096094208 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚428344277631488 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚5683.866632255 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚422264321808384 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚461083500119040 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242677869 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚61631998 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 117441024 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚151 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844{ã#ÑÎäÊT MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚151 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚96393369 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚467462387042304 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 6058062747730 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 105627062 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 105627060 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚422264321808384 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6079955823104 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242661890 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461083524059136 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚417249650888186 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6080448653312 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚428344788464640 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 242661890 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚422264339811328 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 137034830 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6080448653312 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34761524 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6378745542144 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚411192096094208 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚428344277631488 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚5683.866632255 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚422264321808384 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚461083500119040 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242677869 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚61631998 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 117441024 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062Ó²ðz–2å- _wandbruntime‚5684  _runtime‚5684àB“V)æÐR RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚96404426 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34765924 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚428381472786944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461121375852544 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 137042132 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6379228109824 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7253 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 105637866 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6057935442746 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚61638502 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 242695977 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6080813779968 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚428381983620096 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 _runtime‚5684.801014604¶3|j[)çÕR I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 242695977 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6080813779968 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚428381983620096 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 _runtime‚5684.801014604 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚96404426 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34765924 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚428381472786944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461121375852544 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 137042132 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6379228109824 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7253 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 105637866 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6057935442746 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚61638502 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 B&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5252103209495544 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15262764692306519 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20434552431106567 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 particle/accuracy_3_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚43.77483282549899 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965049028396606 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08603008836507797 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6259391903877258 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20637184381484985 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.288949191570282 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5786448121070862 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.558025062084198 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10153257101774216 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.26574620604515076 particle/accuracy_4_of_5‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872550964355469 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.414930522441864 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2752988636493683 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.361155241727829 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14218944311141968 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17474746704101562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5695128440856934 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904612898826599 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02162274532020092 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_7‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28397470712661743 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.518041729927063 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16725145280361176 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.42260676622390747 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 jet/accuracy_2_of_7‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.265259712934494 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5935395359992981 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4995374381542206 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9727503061294556 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15871991217136383 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32704341411590576 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_3‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5584357380867004 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 _step‚597 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9817934632301331 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08149001747369766 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11419796198606491 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 jet/accuracy_3_of_6‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44320327043533325 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2361111044883728 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976053237915039 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8198176622390747 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5356990098953247 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19317592680454254 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.014490527100861073 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46919864416122437 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015032660216093063 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5604457259178162 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5900940299034119 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6082717776298523 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15079237520694733 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5019176602363586 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5902777910232544 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17409871518611908 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3990327715873718 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03305701166391373 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07971469312906265 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3691781163215637 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6895406246185303 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1023791953921318 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9749232530593872 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985368549823761 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2696099281311035 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.29629629850387573 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2696290612220764 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4625554382801056 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01646510511636734 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.48218390345573425 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.549184262752533 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16963814198970795 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34702634811401367 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14861929416656494 particle/accuracy_1_of_4‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22324585914611816 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.5110442042350769 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6089285612106323 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _runtime‚5697.468349033 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4488089978694916 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21318849921226501 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05708407238125801 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5302083492279053 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5551753044128418 jet/accuracy_5_of_6‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4485839009284973 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876083731651306 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.171909362077713 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9772376418113708 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5077439546585083 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011303628794848919 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4353070557117462 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.553331196308136 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981234669685364 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09424546360969543 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.027358490973711014 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04174854978919029 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 particle/accuracy_5_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46176016330718994 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666865348816 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5644230842590332 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1866520196199417 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39676716923713684 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23980717360973358 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5811412334442139 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4089146852493286 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44466233253479004 % _timestamp‚1.7884536059114778e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022270116955041885 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4441077411174774 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_1_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22892461717128754 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2067917138338089 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1283164918422699 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2264367640018463 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 jet/accuracy_1_of_6‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39277198910713196 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12978430092334747 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2510540783405304 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5884960889816284 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2621021866798401 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978158473968506 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5367475152015686 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.243954598903656 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6495094299316406 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990009605884552 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5145003199577332 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9649122953414917 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03269314765930176 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.004166666883975267 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5890803933143616 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023894313722848892 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037811607122421265 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8684210777282715 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_2_of_5‚0.4931836426258087 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18906165659427643 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491228342056274 jet/accuracy_4_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16453513503074646 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3538583517074585 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.504865288734436 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15842798352241516 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36752182245254517 jet/accuracy_3_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5305068492889404 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975623607635498 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943597912788391 *jet/accuracy_1_of_4‚0.9235148429870605 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1485074758529663 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5126436352729797 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833336114883423 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0042484113946557045 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5338304042816162 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5385480523109436 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10288066416978836 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8568018674850464 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBtôvf¶Â_purity‚0.9900921583175659 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799255132675171 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5499843955039978 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11959875375032425 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.194144144654274 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11876893788576126 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18906165659427643 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3943389654159546 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20172415673732758 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9702380895614624 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11575780808925629 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333230018616 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5198268294334412 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18741770088672638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9793828129768372 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9838401675224304 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5474527478218079 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16023357212543488 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.309817373752594 ,jet/accuracy_4_of_5‚0.017921147868037224 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5324451327323914 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6034301519393921 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5152032971382141 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_2‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1537802815437317 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3418154716491699 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4426426887512207 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5770408511161804 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14106020331382751 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.361111044883728 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04734831675887108 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8844849467277527 +particle/specificity‚0.9532067179679871 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 epoch‚5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5874943137168884 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3430555760860443 7validation_average_jet_accuracy‚0.45347848534584045 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16049647331237793 trainer/global_step‚7253Sð¼›0l쑦 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2510540783405304 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5884960889816284 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2621021866798401 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978158473968506 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5367475152015686 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.243954598903656 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6495094299316406 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990009605884552 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5145003199577332 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9649122953414917 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03269314765930176 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.004166666883975267 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5890803933143616 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023894313722848892 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037811607122421265 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8684210777282715 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_2_of_5‚0.4931836426258087 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18906165659427643 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5679011940956116 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491228342056274 jet/accuracy_4_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16453513503074646 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3538583517074585 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.504865288734436 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15842798352241516 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36752182245254517 jet/accuracy_3_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5305068492889404 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975623607635498 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943597912788391 *jet/accuracy_1_of_4‚0.9235148429870605 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1485074758529663 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5126436352729797 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833336114883423 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0042484113946557045 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5338304042816162 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5385480523109436 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10288066416978836 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8568018674850464 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9900921583175659 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799255132675171 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5499843955039978 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11959875375032425 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2420634776353836 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.194144144654274 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11876893788576126 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18906165659427643 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3943389654159546 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20172415673732758 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9702380895614624 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11575780808925629 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5733333230018616 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5198268294334412 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18741770088672638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9793828129768372 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9838401675224304 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5474527478218079 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16023357212543488 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.309817373752594 ,jet/accuracy_4_of_5‚0.017921147868037224 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5324451327323914 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6034301519393921 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5152032971382141 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_1_of_2‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1537802815437317 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3418154716491699 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4426426887512207 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5770408511161804 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14106020331382751 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.361111044883728 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04734831675887108 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8844849467277527 +particle/specificity‚0.9532067179679871 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 epoch‚5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5874943137168884 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3430555760860443 7validation_average_jet_accuracy‚0.45347848534584045 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16049647331237793 trainer/global_step‚7253 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5560882687568665 +jet/accuracy_2_of_4‚0.39155009388923645 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7745792269706726 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20276789367198944 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5489040017127991 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882311224937439 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6028481721878052 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6018944978713989 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5499843955039978 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3358324468135834 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5746914744377136 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 *jet/accuracy_1_of_5‚0.9581860899925232 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.593182384967804 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004445995204150677 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_6‚0.057692307978868484 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20543161034584045 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.557397723197937 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9853209853172302 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4714425504207611 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4753086566925049 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012055933475494385 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5669392943382263 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030733108520508 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11622429639101028 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20216788351535797 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5427692532539368 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5252103209495544 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15262764692306519 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20434552431106567 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 particle/accuracy_3_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚43.77483282549899 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965049028396606 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08603008836507797 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6259391903877258 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20637184381484985 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.288949191570282 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5786448121070862 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.558025062084198 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10153257101774216 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.26574620604515076 particle/accuracy_4_of_5‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872550964355469 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.414930522441864 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2752988636493683 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.361155241727829 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14218944311141968 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17474746704101562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5695128440856934 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904612898826599 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02162274532020092 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_7‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28397470712661743 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.518041729927063 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16725145280361176 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.42260676622390747 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 jet/accuracy_2_of_7‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.265259712934494 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5935395359992981 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4995374381542206 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9727503061294556 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15871991217136383 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32704341411590576 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_3‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5584357380867004 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 _step‚597 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9817934632301331 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08149001747369766 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11419796198606491 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 jet/accuracy_3_of_6‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44320327043533325 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2361111044883728 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976053237915039 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8198176622390747 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5356990098953247 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19317592680454254 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.014490527100861073 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46919864416122437 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015032660216093063 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5604457259178162 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5900940299034119 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6082717776298523 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15079237520694733 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5019176602363586 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5902777910232544 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17409871518611908 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3990327715873718 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03305701166391373 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07971469312906265 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ,&Purityì®´ìè&/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3691781163215637 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6895406246185303 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1023791953921318 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9749232530593872 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985368549823761 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2696099281311035 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.29629629850387573 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2696290612220764 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4625554382801056 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01646510511636734 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.48218390345573425 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.549184262752533 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16963814198970795 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34702634811401367 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14861929416656494 particle/accuracy_1_of_4‚1 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22324585914611816 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.5110442042350769 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6089285612106323 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _runtime‚5697.468349033 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4488089978694916 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21318849921226501 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05708407238125801 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5302083492279053 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5551753044128418 jet/accuracy_5_of_6‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4485839009284973 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876083731651306 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.171909362077713 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9772376418113708 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5077439546585083 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011303628794848919 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4353070557117462 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.553331196308136 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981234669685364 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09424546360969543 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.027358490973711014 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04174854978919029 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 particle/accuracy_5_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46176016330718994 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4791666865348816 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5644230842590332 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1866520196199417 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39676716923713684 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23980717360973358 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5811412334442139 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4089146852493286 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44466233253479004 % _timestamp‚1.7884536059114778e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022270116955041885 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4441077411174774 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_1_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22892461717128754 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2067917138338089 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1283164918422699 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2264367640018463 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 jet/accuracy_1_of_6‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39277198910713196 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12978430092334747ÕÖDNœí‚(:‘ èÅæÔšîÇÕ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚76.133' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.226599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯½·‘=î‚(:² èÅæÔµýÒØ disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 185.75390625 disk.dm-1.out‚ 219.6640625) disk./.usagePercent‚74.50972176322915 disk.sda3.out‚ 219.6640625" disk.nvme0n1.out‚ 24138.7734375 disk.dm-0.in‚ 185.75390625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.759765625! disk.rbd2.out‚130.96630859375 network.sent‚ 1.7065199e+07 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875$ disk.nvme0c0n1.out‚ 24138.7734375 disk.sda2.out‚0 disk.rbd1.in‚0# memory_percent‚91.5867805480957 cpu‚25.38295148383789 disk.sda.out‚ 219.6640625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-0.out‚ 24138.7734375 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875' proc.memory.availableMB‚ 1378.421875" disk.nvme0c0n1.in‚ 185.75390625" proc.memory.rssMB‚ 2924.6796875* proc.memory.percent‚17.850828170776367 network.recv‚ 2.057263e+06% disk./.usageGB‚1110.2486991882324 disk.rbd3.in‚0 disk.rbd2.in‚0³ €H’ï‚(:‡ ÷ÅæÔ½¤¦× gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.891 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í”%Hð‚(:½ ÷ÅæÔ‡²„Ú( disk.nvme0c0n1.out‚24709.22314453125 disk.nvme0n1.in‚ 185.75390625 disk.drbd1777.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚24709.22314453125 network.recv‚ 2.062794e+06 disk.sda.in‚ 0.01171875 disk.sda.out‚ 220.30078125 disk.sda3.in‚ 0.01171875! disk.rbd2.out‚131.13427734375& disk.nvme0n1.out‚24709.22314453125 disk.dm-1.in‚ 0.01171875! proc.memory.rssMB‚ 2875.953125) disk./.usagePercent‚74.50971433901067 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 185.75390625 disk.dm-1.out‚ 220.30078125* proc.memory.percent‚17.553424835205078% disk./.usageGB‚1110.2485885620117 disk.sda3.out‚ 220.30078125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚1426.125 cpu‚25.38154611089407 disk.drbd1777.in‚0 disk.sda2.out‚0$ memory_percent‚91.29562377929688 disk.rbd0.in‚0 disk.dm-0.in‚ 185.75390625 network.sent‚ 1.718835e+07 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda1.out‚0öoN žñ‚(:“ †ÆæÔÕû†Ö gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.285399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àã\Iò‚(:¾ †ÆæÔÑ¥ÙØ" disk.nvme0c0n1.in‚ 185.75390625 disk.drbd1777.out‚0 disk.rbd0.out‚0! disk.rbd2.out‚132.70849609375( disk.nvme0c0n1.out‚25279.93505859375 disk.nvme0n1.in‚ 185.75390625& disk.nvme0n1.out‚25279.93505859375 disk.drbd1777.in‚0* proc.memory.percent‚17.553424835205078 cpu‚25.38073781887528) disk./.usagePercent‚74.50958684725862 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875! proc.memory.rssMB‚ 2875.953125 proc.cpu.threads‚17 network.recv‚ 2.067164e+06 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 221.1484375 disk.rbd0.in‚0# disk.dm-0.out‚25279.93505859375 disk.dm-1.out‚ 221.1484375" memory_percent‚91.294264793396 disk.rbd1.out‚ 2.759765625 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 221.1484375 disk.dm-0.in‚ 185.75390625 network.sent‚ 1.7194487e+07 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1426.34765625% disk./.usageGB‚1110.2466888427734 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0Žž02ó- _wandbruntime‚5734  _runtime‚5734(…qÈÇôÁ trainer/global_step‚7299 % _timestamp‚1.7884536429568448e+09 _runtime‚5734.511424299 _step‚598 & lr-AdamW/pg1‚7.496427483284684e-05 & lr-AdamW/pg2‚7.496427483284684e-05€ùù ÌõÆ & lr-AdamW/pg1‚7.496427483284684e-05 & lr-AdamW/pg2‚7.496427483284684e-05 trainer/global_step‚7299 % _timestamp‚1.7884536429568448e+09 _runtime‚5734.511424299 _step‚598Ö®ï2ö- _wandbruntime‚5734  _runtime‚5734<•í¢å'÷ßO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62090614 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470810877729280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6121796554752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452917248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106409064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422648679424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928ØÉÍGøäO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚470810877729280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6121796554752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6122452917248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425315828959744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106409064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422648679424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62090614 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784×' CF2ù- _wandbruntime‚5735  _runtime‚5735¤6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131601 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97131145 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚600 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106423080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425372181998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100070897372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137988238 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431494797981184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470873200859648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431384 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106423082 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106423082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚7299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873388410368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414211798732800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244431384 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106423082 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106423082 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚7299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚470873388410368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138008302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414211798732800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131601 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97131145 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚600 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464449694320640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106423080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425372181998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100070897372 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137988238 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431494797981184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470873200859648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473936096590336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97751342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35251914 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107109866 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97750926 XJDeviceStatsMonitor.on_train_batch_start/requested_byteU¿R½Ÿs.small_pool.current‚ 654437756 _runtime‚ 5770.70880518 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428133498905088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473936279960064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138915874 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782360576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884536791541238e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005700 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138895836 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7349 _step‚603 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0pÓ5é'ˆãO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7349 _step‚603 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428133480902144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6163438723072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473936096590336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416902084090880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467470580466176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138915874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97751342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35251914 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107109866 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97750926 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚ 5770.70880518 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428133498905088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428133498905088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107109866 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473936279960064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138915874 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6162782360576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62499014 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416902101130240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107109864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 % _timestamp‚1.7884536791541238e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246005700 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138895836ÛMbYB2‰- _wandbruntime‚5771  _runtime‚5771bv§ï&ŠéM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423098565935544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5771.401812394 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038170 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97763524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998615150080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62507180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458220 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35256344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35256778 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884536798466136e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934354 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97763960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601788416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚604 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6466383946240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428189851943936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434354110108160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236780 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944,mùÚ4‹îM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138914290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6466383946240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428189851943936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434354110108160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6140906236780 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467532231203840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423098565935544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62507182 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5771.401812394 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246038170 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMont³Àitor.on_train_batch_end/active.large_pool.freed‚ 107123880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97763524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998615150080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6141560674744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62507180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚423097894458220 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35256344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35256778 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884536798466136e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934354 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97763960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6163601788416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467532255143936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚604 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066ÜÑûò2Œ- _wandbruntime‚5771  _runtime‚5771°F$WÑ _step‚605 /loss/FBt4/assignment_loss‚0.578362275075051 1loss/SRqqt2/detection_loss‚1.2902934687083651 % _timestamp‚1.7884536798491974e+09 0loss/FRt2/assignment_loss‚1.9313784165815873 1loss/FBt2/assignment_loss‚0.10900934404890181 /loss/FRt1/detection_loss‚2.8739852656920752 .loss/FRt3/detection_loss‚2.093700739572633 /loss/FBt2/detection_loss‚0.6458364935963237 0loss/FBt3/assignment_loss‚0.4086716929540181 0loss/FRt1/assignment_loss‚2.3074414590994516 2loss/SRqqt1/assignment_loss‚1.2928693061857892 2loss/SRqqt3/assignment_loss‚1.6167236291894727 2loss/SRqqt4/assignment_loss‚1.5211257844260244 /loss/FRt4/detection_loss‚3.3669846114658175 _runtime‚5771.403579263 2loss/SRqqt2/assignment_loss‚1.0225673408151186 1loss/FBt1/assignment_loss‚0.18600821109094148 epoch‚6 trainer/global_step‚7349 0loss/SRqqt4/detection_loss‚2.485216173258695 -loss/FBt3/detection_loss‚2.13838673554934 /loss/FBt4/detection_loss‚3.0149499848484993 /loss/FBt1/detection_loss‚2.4358465304741492 %loss/total_loss‚43.31097935832659 .loss/FRt3/assignment_loss‚3.01210407828385 /loss/FRt4/assignment_loss‚3.758475962139311 /loss/FRt2/detection_loss‚1.7396237997168844 1loss/SRqqt1/detection_loss‚1.7966723946625727 /loss/SRqqt3/detection_loss‚1.68474566089172K\ŽÖ /loss/FRt2/detection_loss‚1.7396237997168844 1loss/SRqqt1/detection_loss‚1.7966723946625727 /loss/SRqqt3/detection_loss‚1.68474566089172 _step‚605 /loss/FBt4/assignment_loss‚0.578362275075051 1loss/SRqqt2/detection_loss‚1.2902934687083651 % _timestamp‚1.7884536798491974e+09 0loss/FRt2/assignment_loss‚1.9313784165815873 1loss/FBt2/assignment_loss‚0.10900934404890181 /loss/FRt1/detection_loss‚2.8739852656920752 .loss/FRt3/detection_loss‚2.093700739572633 /loss/FBt2/detection_loss‚0.6458364935963237 0loss/FBt3/assignment_loss‚0.4086716929540181 0loss/FRt1/assignment_loss‚2.3074414590994516 2loss/SRqqt1/assignment_loss‚1.2928693061857892 2loss/SRqqt3/assignment_loss‚1.6167236291894727 2loss/SRqqt4/assignment_loss‚1.5211257844260244 /loss/FRt4/detection_loss‚3.3669846114658175 _runtime‚5771.403579263 2loss/SRqqt2/assignment_loss‚1.0225673408151186 1loss/FBt1/assignment_loss‚0.18600821109094148 epoch‚6 trainer/global_step‚7349 0loss/SRqqt4/detection_loss‚2.485216173258695 -loss/FBt3/detection_loss‚2.13838673554934 /loss/FBt4/detection_loss‚3.0149499848484993 /loss/FBt1/detection_loss‚2.4358465304741492 %loss/total_loss‚43.31097935832659 .loss/FRt3/assignment_loss‚3.01210407828385 /loss/FRt4/assignment_loss‚3.758475962139311Ýé+B¾ž‚(:“ ³ÆæÔ°ÕÄÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.115399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m=ž‰I‚(:¾ ³ÆæÔ´ÈùØ disk.drbd1777.out‚0 disk.rbd1.out‚ 2.908203125$ memory_percent‚91.29619598388672 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 185.75390625 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda3.out‚ 222.96484375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 185.75390625 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 222.96484375 disk.sda1.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚136.25146484375 cpu‚25.378126665975746 network.recv‚ 2.081727e+06 disk.rbd3.out‚0 disk.sda1.in‚0 network.sent‚ 1.7314183e+07% disk./.usageGB‚1110.2474555969238' disk.nvme0c0n1.out‚25378.4306640625 disk.sda2.out‚0 disk.sda.in‚ 0.01171875* proc.memory.percent‚17.553424835205078% disk.nvme0n1.out‚25378.4306640625 disk.drbd1777.in‚0 disk.sda.out‚ 222.96484375 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0& proc.memory.availableMB‚ 1426.03125 disk.dm-0.in‚ 185.75390625" disk.dm-0.out‚25378.4306640625! proc.memory.rssMB‚ 2875.953125) disk./.usagePercent‚74.50963830477299—H:“‘‚(:ˆ ÂÆæÔã™ÚÕ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æUÀO’‚(:Ä ÂÆæÔLJÙ network.recv‚ 2.087923e+06 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚136.25146484375 disk.dm-0.in‚ 185.75390625 network.sent‚ 1.7422804e+07) disk./.usagePercent‚74.50963574469766 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚25476.27880859375 disk.rbd3.out‚0& disk.nvme0n1.out‚25476.27880859375 disk.sda2.in‚0( proc.memory.availableMB‚ 1426.4921875 cpu‚25.377279304731488% disk./.usageGB‚1110.2474174499512 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda.out‚ 223.55859375! proc.memory.rssMB‚ 2875.953125 disk.sda3.out‚ 223.55859375 disk.nvme0n1.in‚ 185.75390625$ memory_percent‚91.29338264465332" disk.nvme0c0n1.in‚ 185.75390625 disk.drbd1777.in‚0 disk.dm-1.out‚ 223.55859375 disk.rbd1.out‚ 2.9404296875 proc.cpu.threads‚17 disk.sda1.out‚0# disk.dm-0.out‚25476.27880859375* proc.memory.percent‚17.5534248352050781]ƒ’“‚(:‡ ÑÆæÔ˶øÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zêTM”‚(: ÑÆæÔÄŒ¬Ø disk.drbd1777.out‚0$ memory_percent‚91.29588603973389) proc.memory.availableMB‚ 1426.08203125 disk.sda1.out‚0) disk./.usagePercent‚74.50963702473533% disk./.usageGB‚1110.2474365234375 disk.drbd1777.in‚0 disk.dm-1.out‚ 224.2265625* proc.memory.percent‚17.553424835205078& disk.nvme0n1.out‚25573.75146484375 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 2.9404296875 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 224.2265625 cpu‚25.376443756117006# disk.dm-0.out‚25573.75146484375 disk.sda2.in‚0 disk.sda3.in‚ 0.01171875! proc.memory.rssMB‚ 2875.953125 disk.rbd2.in‚0! disk.rbd2.out‚137.98974609375 disk.sda.out‚ 224.2265625 disk.sda1.in‚0 network.sent‚ 1.7428411e+07 network.recv‚ 2.091992e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 185.75390625( disk.nvme0c0n1.out‚25573.75146484375 disk.nvme0n1.in‚ 185.75390625 disk.rbd3.out‚0 disk.dm-0.in‚ 185.75390625 proc.cpu.threads‚17s²¥2•- _wandbruntime‚5806  _runtime‚5806ËÒ2Ç–Á & lr-AdamW/pg2‚7.235808222848948e-05 trainer/global_step‚7399 % _timestamp‚1.7884537154075043e+09 _runtime‚5806.961949289 _step‚606 & lr-AdamW/pg1‚7.235808222848948e-05 8`áÌ—Æ & lr-AdamW/pg1‚7.235808222848948e-05 & lr-AdamW/pg2‚7.235808222848948e-05 trainer/global_step‚7399 % _timestamp‚1.7884537154075043e+09 _runtime‚5806.961949289 _step‚606ÞñíþÛ2˜- _wandbruntime‚5806  _runtime‚5806|êiïä'™ÞO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107810666 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6180925141224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139821888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6204424528896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6203768166400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430951132844544 _step‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107810664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98384814 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470553117349376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203768166400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477061773422080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚5806.962858352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98384374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107810666 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35476962 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35477400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6508632132608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788453715408259e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430951132844544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419647290618880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6181579578980 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575376384 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6508632132608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788453715408259e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430951132844544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430951150847488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425828215760104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419647290618880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6181579578980 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437155575376384 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477061773422080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚5806.962858352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98384374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107810666 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35476962 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35477400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitorY¥®%.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98397022 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860406 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35481891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576188 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7399 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860406 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124095368704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509495086592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073477632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6205243970048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587594240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509327281664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884537160882883e+09 _runtime‚ 5807.6432408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35481442 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124287113728 _step‚608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98397473Š\Ôoò&ìM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124287113728 _step‚608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98397473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107824682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98397022 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139860406 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35481891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576188 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7399 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860406 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124095368704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509495086592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437212073477632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470614768087040 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107824682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6205243970048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6204587594240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6509327281664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884537160882883e+09 _runtime‚ 5807.6432408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35481442 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872àɬ\2ž- _wandbruntime‚5807  _runtime‚5807¸‘NYŸÓ 0loss/FRt3/assignment_loss‚3.6068323401154063 0loss/FBt4/assignment_loss‚0.3611005789862247 /loss/FRt2/detection_loss‚1.9325756385530297 1loss/SRqqt4/detection_loss‚2.1597640335559847 0loss/FRt1/assignment_loss‚2.8549260199069977 0loss/FRt2/assignment_loss‚1.4086963785322089 1loss/SRqqt1/assignment_loss‚1.389737350696867 2loss/SRqqt3/assignment_loss‚1.2202419111866276 1loss/FBt3/assignment_loss‚0.07521826185984419 /loss/FRt4/assignment_loss‚3.634168948446001 2loss/SRqqt2/assignment_loss‚1.2188898885238266 /loss/FRt3/detection_loss‚2.1002519912407047 1loss/SRqqt1/detection_loss‚1.9722434543750502 /loss/FBt1/detection_loss‚2.2984299527274237 /loss/FBt4/detection_loss‚2.1230560950934887 1loss/SRqqt2/detection_loss‚0.9811520277271195 /loss/FBt1/assignment_loss‚0.306073186192609 0loss/FBt2/assignment_loss‚0.1379759251607303 trainer/global_step‚7399 _step‚609 2loss/SRqqt4/assignment_loss‚1.7479782050475479 .loss/FBt3/detection_loss‚3.496879061063131 $ _timestamp‚1.788453716090046e+09 .loss/FRt4/detection_loss‚2.794731146948678 1loss/SRqqt3/detection_loss‚1.8670419390712465 $loss/total_loss‚42.5601623848941 epoch‚6 /loss/FRt1/detection_loss‚2.3608816188006174 /loss/FBt2/detection_loss‚0.5113164310827331 _runtime‚5807.6443431272¹^ Ø 0loss/FRt1/assignment_loss‚2.8549260199069977 0loss/FRt2/assignment_loss‚1.4086963785322089 1loss/SRqqt1/assignment_loss‚1.389737350696867 2loss/SRqqt3/assignment_loss‚1.2202419111866276 1loss/FBt3/assignment_loss‚0.07521826185984419 /loss/FRt4/assignment_loss‚3.634168948446001 2loss/SRqqt2/assignment_loss‚1.2188898885238266 /loss/FRt3/detection_loss‚2.1002519912407047 1loss/SRqqt1/detection_loss‚1.9722434543750502 /loss/FBt1/detection_loss‚2.2984299527274237 /loss/FBt4/detection_loss‚2.1230560950934887 1loss/SRqqt2/detection_loss‚0.9811520277271195 /loss/FBt1/assignment_loss‚0.306073186192609 0loss/FBt2/assignment_loss‚0.1379759251607303 trainer/global_step‚7399 _step‚609 2loss/SRqqt4/assignment_loss‚1.7479782050475479 .loss/FBt3/detection_loss‚3.496879061063131 $ _timestamp‚1.788453716090046e+09 .loss/FRt4/detection_loss‚2.794731146948678 1loss/SRqqt3/detection_loss‚1.8670419390712465 $loss/total_loss‚42.5601623848941 epoch‚6 /loss/FRt1/detection_loss‚2.3608816188006174 /loss/FBt2/detection_loss‚0.5113164310827331 _runtime‚5807.644343127 0loss/FRt3/assignment_loss‚3.6068323401154063 0loss/FBt4/assignment_loss‚0.3611005789862247 /loss/FRt2/detection_loss‚1.9325756385530297 1loss/SRqqt4/detection_loss‚2.1597640335559847áÌ™¡‚(:’ àÆæÔšØïÕ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.710000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ü|ÆM¢‚(: àÆæÔ§òºÙ$ memory_percent‚91.29757881164551 cpu‚25.375596107366203 disk.nvme0n1.in‚ 185.75390625 disk.drbd1777.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1425.8046875 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚25574.5986328125 network.recv‚ 2.096606e+06 disk.sda.out‚ 224.69140625 disk.sda3.out‚ 224.69140625" disk.nvme0c0n1.in‚ 185.75390625% disk.nvme0n1.out‚25574.5986328125 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17! disk.rbd2.out‚137.98974609375 network.sent‚ 1.7434701e+07% disk./.usageGB‚1110.2473068237305 disk.rbd0.out‚0 disk.rbd1.out‚ 3.00341796875 disk.sda.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.553424835205078) disk./.usagePercent‚74.50962832047917 disk.sda3.in‚ 0.01171875" disk.dm-0.out‚25574.5986328125 disk.sda2.out‚0! proc.memory.rssMB‚ 2875.953125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 185.75390625 disk.dm-1.out‚ 224.69140625 disk.rbd1.in‚0~:B£‚(:’ ïÆæÔ©—€Ö gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.19' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.037999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ЯʗM¤‚(: ïÆæÔÓ‰òØ disk.rbd1.out‚ 3.00341796875 disk.rbd3.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚25672.0009765625% disk.nvme0n1.out‚25672.0009765625* proc.memory.percent‚17.553424835205078 proc.cpu.threads‚17 network.recv‚ 2.102159e+06 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 185.75390625 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 225.33984375 disk.rbd3.in‚0 disk.sda3.out‚ 225.33984375 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.37476781963092) disk./.usagePercent‚74.50962960051683 disk.sda.out‚ 225.33984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2875.953125 network.sent‚ 1.7542692e+07! disk.rbd2.out‚139.60693359375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 185.75390625" disk.dm-0.out‚25672.0009765625 disk.sda2.out‚0% disk./.usageGB‚1110.2473258972168 disk.dm-0.in‚ 185.75390625$ memory_percent‚91.29612445831299) proc.memory.availableMB‚ 1426.04296875šžvú2¥- _wandbruntime‚5843  _runtime‚5843£^µDŦ¿ % _timestamp‚1.7884537514862905e+09 _runtime‚5843.040666108 _step‚610 % lr-AdamW/pg1‚7.10610348117076e-05 % lr-AdamW/pg2‚7.10610348117076e-05 trainer/global_step‚7449Ûô)ŽÊ§Ä trainer/global_step‚7449 % _timestamp‚1.7884537514862905e+09 _runtime‚5843.040666108 _step‚610 % lr-AdamW/pg1‚7.10610348117076e-05 % lr-AdamW/pg2‚7.10610348117076e-05âj:„2¨- _wandbruntime‚5843  _runtime‚5843ÏPÉN©ÞO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538759168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422392497146880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480187177522688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35702837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213124608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 $ _timestamp‚1.788453751486961e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108511466 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 LFDeviceStatsMonitor.o¿:&– n_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433768802789888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245410334720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63315814 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551499350016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5843.041676532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35703255 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433768802789888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245410334720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108511464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63315814 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551499350016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5843.041676532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35703255 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480187177522688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627512 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414ãl£Î2«- _wandbruntime‚5843  _runtime‚5843îœãÊï&¬éM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473697328910336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35707272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99031252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚612 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 % _timestamp‚1.7884537521609871e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440071385604608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766394 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140786458 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473697304970240 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249500493312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229775872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140766394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573400064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35707711 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552363328000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99031693 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422447401277440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223231353560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140786458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573400064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249692238336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736;L_=#­îM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245573400064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35707711 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552363328000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99031693 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422447401277440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223231353560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140786458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573400064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249692238336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473697328910336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63323980 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35707272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99031252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚612 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 % _timestamp‚1.7884537521609871e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440071385604608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433825137825792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140766394 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108525482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batc ݜͷh_end/allocation.small_pool.allocated‚ 140786458 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473697304970240 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108525480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480249500493312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229775872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140766394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0äð„2®- _wandbruntime‚5843  _runtime‚5843ºý6Ð^¯Ø _step‚613 epoch‚6 0loss/FBt3/assignment_loss‚0.3718440576456487 /loss/FRt1/detection_loss‚2.7052779713502297 /loss/FRt4/detection_loss‚2.7163880705833434 1loss/SRqqt1/detection_loss‚1.9240119377983378 2loss/SRqqt4/assignment_loss‚1.6041452209465206 /loss/FBt2/detection_loss‚0.5051334766315425 0loss/FRt1/assignment_loss‚3.0031713980894823 1loss/FBt2/assignment_loss‚0.18003499441231094 0loss/FRt4/assignment_loss‚3.4181782921155293 1loss/FBt4/assignment_loss‚0.43007871818762006 /loss/FRt2/detection_loss‚1.3555681930213679 0loss/SRqqt4/detection_loss‚2.371902803890407 0loss/FRt2/assignment_loss‚1.1794720467973927 2loss/SRqqt2/assignment_loss‚0.9556168743284114 1loss/SRqqt2/detection_loss‚0.8478740241322944 1loss/SRqqt3/detection_loss‚1.9680951015622008 0loss/FRt3/assignment_loss‚3.7994307049533775 1loss/SRqqt1/assignment_loss‚1.106378532153495 .loss/FBt4/detection_loss‚2.332783724580492 _runtime‚5843.717704926 0loss/FBt1/assignment_loss‚0.4499031178004397 trainer/global_step‚7449 % _timestamp‚1.7884537521633952e+09 %loss/total_loss‚41.46507065581862 2loss/SRqqt3/assignment_loss‚1.5949896223770053 /loss/FRt3/detection_loss‚2.0344235640868806 /loss/FBt1/detection_loss‚2.3992587668555125 /loss/FBt3/detection_loss‚2.2111094415187837Ô³Ic°Ý 1loss/FBt2/assignment_loss‚0.18003499441231094 0loss/FRt4/assignment_loss‚3.4181782921155293 1loss/FBt4/assignment_loss‚0.43007871818762006 /loss/FRt2/detection_loss‚1.3555681930213679 0loss/SRqqt4/detection_loss‚2.371902803890407 0loss/FRt2/assignment_loss‚1.1794720467973927 2loss/SRqqt2/assignment_loss‚0.9556168743284114 1loss/SRqqt2/detection_loss‚0.8478740241322944 1loss/SRqqt3/detection_loss‚1.9680951015622008 0loss/FRt3/assignment_loss‚3.7994307049533775 1loss/SRqqt1/assignment_loss‚1.106378532153495 .loss/FBt4/detection_loss‚2.332783724580492 _runtime‚5843.717704926 0loss/FBt1/assignment_loss‚0.4499031178004397 trainer/global_step‚7449 % _timestamp‚1.7884537521633952e+09 %loss/total_loss‚41.46507065581862 2loss/SRqqt3/assignment_loss‚1.5949896223770053 /loss/FRt3/detection_loss‚2.0344235640868806 /loss/FBt1/detection_loss‚2.3992587668555125 /loss/FBt3/detection_loss‚2.2111094415187837 _step‚613 epoch‚6 0loss/FBt3/assignment_loss‚0.3718440576456487 /loss/FRt1/detection_loss‚2.7052779713502297 /loss/FRt4/detection_loss‚2.7163880705833434 1loss/SRqqt1/detection_loss‚1.9240119377983378 2loss/SRqqt4/assignment_loss‚1.6041452209465206 /loss/FBt2/detection_loss‚0.5051334766315425 0loss/FRt1/assignment_loss‚3.0031713980894823åa(|ž±‚(:“ þÆæÔѸ½Õ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.861' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.172199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t(UL²‚(:Á þÆæÔปØ disk.sda1.out‚0! disk.rbd2.out‚139.60693359375 disk.sda2.in‚0( proc.memory.availableMB‚ 1425.9921875 network.recv‚ 2.106318e+06) disk./.usagePercent‚74.50966262548874 disk.rbd2.in‚0& disk.nvme0n1.out‚25769.64501953125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 225.69140625 disk.drbd1777.out‚0 disk.dm-1.out‚ 225.69140625 disk.rbd1.out‚ 3.06640625 disk.rbd3.in‚0! proc.memory.rssMB‚ 2875.953125 disk.nvme0n1.in‚ 185.75390625 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0* proc.memory.percent‚17.553424835205078 network.sent‚ 1.7548985e+07( disk.nvme0c0n1.out‚25769.64501953125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 185.75390625# disk.dm-0.out‚25769.64501953125 disk.sda2.out‚0$ memory_percent‚91.29643440246582 cpu‚25.373987361805778$ disk./.usageGB‚1110.247817993164 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 185.75390625 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 225.69140625k«è§“³‚(:ˆ ÇæÔÄÕòÕ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.029' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T~\1L´‚(:Á ÇæÔãó Ú disk.rbd2.out‚ 141.291015625 disk.dm-1.in‚ 0.01171875% disk./.usageGB‚1110.2478370666504 disk.rbd1.out‚ 3.06640625 disk.sda.out‚ 226.72265625 disk.sda1.out‚0 disk.sda3.out‚ 226.72265625# disk.dm-0.out‚25770.03466796875 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚25770.03466796875$ memory_percent‚91.29650592803955) proc.memory.availableMB‚ 1425.98046875* proc.memory.percent‚17.553424835205078 cpu‚25.373143991673494 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 185.75390625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 185.75390625 disk.sda2.in‚0 disk.dm-1.out‚ 226.72265625( disk.nvme0c0n1.out‚25770.03466796875 disk.nvme0n1.in‚ 185.75390625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2875.953125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.7657071e+07 network.recv‚ 2.112223e+06) disk./.usagePercent‚74.50966390552642 disk.rbd0.in‚0çÚ:…2µ- _wandbruntime‚5879  _runtime‚5879ù-VǶÁ & lr-AdamW/pg1‚6.976838067524785e-05 & lr-AdamW/pg2‚6.976838067524785e-05 trainer/global_step‚7499 % _timestamp‚1.7884537878151834e+09 _runtime‚5879.369550907 _step‚614%­†]Ì·Æ % _timestamp‚1.7884537878151834e+09 _runtime‚5879.369550907 _step‚614 & lr-AdamW/pg1‚6.976838067524785e-05 & lr-AdamW/pg2‚6.976838067524785e-05 trainer/global_step‚7499æe.vû2¸- _wandbruntime‚5879  _runtime‚5879oàL å'¹ßO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63724212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436586436729344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141694030 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476718191115776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436586454732288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63724214 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483312491377664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425137720714240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141694030 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312681038848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431400299494920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚7499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250886256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6594300261888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5879.370564227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141673992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141673992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344 % _timestamp‚1.7884537878159125e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109212266 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99652990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚476718215055872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109212266 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35928330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109212264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99652542 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436586436729344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141694030 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476718191115776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436586454732288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63724214 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483312491377664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425137720714240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141694030 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312681038848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431400299494920 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚7499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250886256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6594300261888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5879.370564227 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141673992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141673992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344 % _timestamp‚1.7884537878159125e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109212266 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99652990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚476718215055872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109212266 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35928330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109212264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99652542 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348974080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250918726 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 trainer/global_step‚7499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109226282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6595153527808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99665217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884537884951997e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436642789768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286559205888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712510 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35933275 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚5880.050132988 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483374995381248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141692446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35932837 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0©wn%ô&¿îM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35932837 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442929348974080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250918726 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 trainer/global_step‚7499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109226282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6595153527808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63732382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99665217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476779865793536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884537884951997e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚436642789768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286559205888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141712510 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35933275 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚5880.050132988 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483374995381248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141692446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64132612 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36154175 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7549 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36153753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437879864832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6326725583872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100286789 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142620082 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6326725583872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434186341362328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6304085597204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814255616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100286365 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pÐÜP4~ool.allocated‚64132614 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884538239137013e+0953`×ì'ÌæO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100286789 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚479800751939072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142620082 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6326725583872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚434186341362328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6304085597204 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814255616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439404088671744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100286365 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64132614 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884538239137013e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64132612 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36154175 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7549 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36153753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486437879864832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6326725583872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424ëo­2Í- _wandbruntime‚5916  _runtime‚5916ÝïÃdñ&ÎëM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618498 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445788661101056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434242733405176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36158664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439460441710592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434242061927852 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 trainer/global_step‚7549 _runtime‚5916.147476194 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚620 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6328201387520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479862402676736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500406488576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6638027751936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚421 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36158245 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100299025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986722304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6327545011712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486500585650688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439460459713536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142638562 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100299446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545011712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884538245923584e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479862402676736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500406488576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6638027751936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚421 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36158245 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100299025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445787986722304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6327545011712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486500585650688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚439460459713536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142638562 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100299446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545011712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884538245923584e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445788661101056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚434242733405176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36158664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439460441710592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434242061927852 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 trainer/global_step‚7549 _runtime‚5916.147476194 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚620 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498ìðG^å2Ð- _wandbruntime‚5916  _runtime‚5916ÖN6YÑÓ 1loss/SRqqt1/assignment_loss‚1.525477892793715 /loss/FRt3/detection_loss‚2.1226763772693564 .loss/FBt1/detection_loss‚2.365912311606937 %loss/total_loss‚43.41552753847877 epoch‚6 1loss/SRqqt2/assignment_loss‚0.939185476662325 /loss/FRt4/detection_loss‚3.3642780826641965 1loss/SRqqt2/detection_loss‚0.9455036438203284 $ _timestamp‚1.788453824594476e+09 0loss/FRt1/assignment_loss‚2.9908364686098965 /loss/FRt1/detection_loss‚2.3793797486207704 /loss/FBt4/detection_loss‚2.7238782346248627 /loss/FRt2/assignment_loss‚1.462694875895977 2loss/SRqqt3/assignment_loss‚1.8328482440494476 2loss/SRqqt4/assignment_loss‚1.8893056548125036 1loss/SRqqt1/detection_loss‚1.8330775055289268 0loss/FBt1/assignment_loss‚0.2631619711497011 1loss/SRqqt3/detection_loss‚1.7732036085204874 /loss/FBt3/detection_loss‚2.0186399654908613 trainer/global_step‚7549 _step‚621 0loss/FRt4/assignment_loss‚3.3935405061795163 1loss/FBt3/assignment_loss‚0.46123850968433544 0loss/FBt4/assignment_loss‚0.8374002327521642 1loss/SRqqt4/detection_loss‚2.4670184174099483 _runtime‚5916.148764141 /loss/FRt3/assignment_loss‚3.272194816307588 /loss/FBt2/assignment_loss‚0.126598440399451 /loss/FRt2/detection_loss‚1.7806563436364133 /loss/FBt2/detection_loss‚0.6468202099890615¾S¨ºÒØ epoch‚6 1loss/SRqqt2/assignment_loss‚0.939185476662325 /loss/FRt4/detection_loss‚3.3642780826641965 1loss/SRqqt2/detection_loss‚0.9455036438203284 $ _timestamp‚1.788453824594476e+09 0loss/FRt1/assignment_loss‚2.9908364686098965 /loss/FRt1/detection_loss‚2.3793797486207704 /loss/FBt4/detection_loss‚2.7238782346248627 /loss/FRt2/assignment_loss‚1.462694875895977 2loss/SRqqt3/assignment_loss‚1.8328482440494476 2loss/SRqqt4/assignment_loss‚1.8893056548125036 1loss/SRqqt1/detection_loss‚1.8330775055289268 0loss/FBt1/assignment_loss‚0.2631619711497011 1loss/SRqqt3/detection_loss‚1.7732036085204874 /loss/FBt3/detection_loss‚2.0186»è†o¤399654908613 trainer/global_step‚7549 _step‚621 0loss/FRt4/assignment_loss‚3.3935405061795163 1loss/FBt3/assignment_loss‚0.46123850968433544 0loss/FBt4/assignment_loss‚0.8374002327521642 1loss/SRqqt4/detection_loss‚2.4670184174099483 _runtime‚5916.148764141 /loss/FRt3/assignment_loss‚3.272194816307588 /loss/FBt2/assignment_loss‚0.126598440399451 /loss/FRt2/detection_loss‚1.7806563436364133 /loss/FBt2/detection_loss‚0.6468202099890615 1loss/SRqqt1/assignment_loss‚1.525477892793715 /loss/FRt3/detection_loss‚2.1226763772693564 .loss/FBt1/detection_loss‚2.365912311606937 %loss/total_loss‚43.41552753847877íƒJ°T“Ó‚(:ˆ ÉÇæÔ˜ŸÉÕ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚105.104' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ËNMEÔ‚(:º ÉÇæÔ˜þ‘Ù disk.drbd1777.out‚0# memory_percent‚91.2984848022461 proc.cpu.threads‚17 disk.sda3.out‚ 229.6328125 disk.rbd2.out‚ 144.521484375 disk.rbd1.out‚ 3.06640625 disk.rbd3.in‚0 disk.dm-0.in‚ 189.34765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 229.6328125 disk.sda1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚26069.34619140625! proc.memory.rssMB‚ 2875.953125 disk.rbd3.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875" disk.nvme0c0n1.in‚ 189.34765625 disk.nvme0n1.in‚ 189.34765625% disk./.usageGB‚1110.2467956542969* proc.memory.percent‚17.553424835205078 cpu‚25.369853779809418 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0) disk./.usagePercent‚74.50959401546957 disk.dm-1.out‚ 229.6328125 disk.drbd1777.in‚0 network.sent‚ 1.7863413e+07 network.recv‚ 2.132625e+06 disk.rbd0.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 1425.65625# disk.dm-0.out‚26069.34619140625( disk.nvme0c0n1.out‚26069.34619140625Z`¥GžÕ‚(:“ ØÇæÔƒ¡¼Õ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.999599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ú.(AÖ‚(:¶ ØÇæÔóÃÛØ disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1425.5078125% disk.nvme0n1.out‚26071.0478515625 disk.sda3.out‚ 230.17578125 disk.rbd2.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.553424835205078" disk.dm-0.out‚26071.0478515625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚26071.0478515625 disk.rbd1.out‚ 3.06640625 cpu‚25.369053287038238 disk.rbd0.in‚0 disk.dm-0.in‚ 189.796875 disk.dm-1.out‚ 230.17578125 disk.sda.in‚ 0.01171875 disk.sda.out‚ 230.17578125 disk.sda1.out‚0% disk./.usageGB‚1110.2469215393066 disk.nvme0c0n1.in‚ 189.796875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.sent‚ 1.7972344e+07 disk.rbd2.out‚ 146.095703125 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0) disk./.usagePercent‚74.50960246371821 disk.nvme0n1.in‚ 189.796875$ memory_percent‚91.29939079284668! proc.memory.rssMB‚ 2875.953125 proc.cpu.threads‚17 network.recv‚ 2.13888e+06„tö2×- _wandbruntime‚5951  _runtime‚5951‹µmÇØÁ % _timestamp‚1.7884538603365977e+09 _runtime‚5951.891023464 _step‚622 & lr-AdamW/pg1‚6.719732882034002e-05 & lr-AdamW/pg2‚6.719732882034002e-05 trainer/global_step‚7599.žPÌÙÆ _runtime‚5951.891023464 _step‚622 & lr-AdamW/pg1‚6.719732882034002e-05 & lr-AdamW/pg2‚6.719732882034002e-05 trainer/global_step‚7599 % _timestamp‚1.7884538603365977e+09î€Íˆ2Ú- _wandbruntime‚5951  _runtime‚5951åù„Eæ'ÛàO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680442909184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563542070272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚5951.892087439 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7599 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143546134 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143526096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36380515 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788453860337377e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380950 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680277188096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Øeë'ÜåO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143546134 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143526096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36380515 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788453860337377e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380950 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442221758617088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680277188096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430628133770240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680442909184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563542070272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚5951.892087439 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7599ï]¤2Ý- _wandbruntime‚5952  _runtime‚5952i÷±îPÞëM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100934639 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489626254314496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6345082933820 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143564614 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442278093652992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 144736256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110627882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚7599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143544550 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298849280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626085638144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100934211 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884538610148075e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6345737371784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_by´E@¡tes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530817536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36385457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6681314754560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368530817536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36385457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6681314754560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110627882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚7599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64549182 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143544550 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298849280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626085638144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100934211 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884538610148075e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442278111655936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6345737371784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101556136 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101555691 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353558016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439758425097144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6723356187136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492689181892608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6408697195520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688985939968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6409353558016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111314666 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472186 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445039392556544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6385756276020 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36606722 _step‚627^«¬çª îæO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101556136 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101555691 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353558016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439758425097144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6723356187136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492689181892608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6408697195520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492688985939968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452148 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 [IDeviceS6j‘BtatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6409353558016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111314666 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472186 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445039392556544 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6385756276020 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36606722 _step‚627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6723184174592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451448764117504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433373340298240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884538965749302e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144452148 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚5988.129547056óÞȹ2ï- _wandbruntime‚5988  _runtime‚5988³5ðñ&ðëM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚5988.799694901 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172999168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884538972444508e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724208420864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101568387 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751684864000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439814145662668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111328680 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚628 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144490666 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101568824 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36611242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751499410432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64957582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144470602 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6385918273228 trainer/global_step‚7649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64957580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6386572711192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36610807 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111328682O‘¹Žö&ñðM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚5988.799694901 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172999168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884538972444508e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724208420864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101568387 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751684864000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439814145662668 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111328680 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚628 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144490666 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101568824 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36611242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751499410432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64957582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144470602 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6385918273228 trainer/global_step‚7649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64957580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6386572711192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36610807 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111328682 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ôíö;_2ò- _wandbruntime‚5988  _runtime‚5988*pe`TóÎ 1loss/SRqqt2/detection_loss‚0.9860820395575054 0loss/SRqqt3/detection_loss‚1.750275054978112 % _timestamp‚1.7884538972466903e+09 0loss/FRt1/assignment_loss‚2.7048795554372997 /loss/FRt3/assignment_loss‚3.005001464994942 2loss/SRqqt2/assignment_loss‚1.0783381241279235 .loss/FRt1/detection_loss‚2.435042746199502 .loss/FRt3/detection_loss‚2.421584881660415 1loss/SRqqt4/detection_loss‚2.0868641897743823 .loss/FBt3/detection_loss‚2.057752900653415 trainer/global_step‚7649 2loss/SRqqt3/assignment_loss‚1.8398245958200956 1loss/FBt1/assignment_loss‚0.00911962376825386 0loss/FBt3/assignment_loss‚0.3881659320452147 1loss/SRqqt4/assignment_loss‚1.853392414529534 1loss/FBt2/assignment_loss‚0.09086066184999585 0loss/FBt4/assignment_loss‚0.4131934553248963 $loss/total_loss‚41.9614842475092 _runtime‚5988.800985303 /loss/FRt2/assignment_loss‚1.362666216763583 2loss/SRqqt1/assignment_loss‚1.4170505294459872 /loss/FRt4/detection_loss‚2.9039453342556953 .loss/FBt4/detection_loss‚2.366731365521749 1loss/SRqqt1/detection_loss‚1.8445120431715623 /loss/FBt2/detection_loss‚0.6088439210786565 /loss/FRt4/assignment_loss‚3.404604744166136 -loss/FRt2/detection_loss‚1.55429771352898 .loss/FBt1/detection_loss‚3.378454738855362 epoch‚6 _step‚629ÌÆ]´YôÓ 0loss/FBt4/assignment_loss‚0.4131934553248963 $loss/total_loss‚41.9614842475092 _runtime‚5988.800985303 /loss/FRt2/assignment_loss‚1.362666216763583 2loss/SRqqt1/assignment_loss‚1.4170505294459872 /loss/FRt4/detection_loss‚2.9039453342556953 .loss/FBt4/detection_loss‚2.366731365521749 1loss/SRqqt1/detection_loss‚1.8445120431715623 /loss/FBt2/detection_loss‚0.6088439210786565 /loss/FRt4/assignment_loss‚3.404604744166136 -loss/FRt2/detection_loss‚1.55429771352898 .loss/FBt1/detection_loss‚3.378454738855362 epoch‚6 _step‚629 1loss/SRqqt2/detection_loss‚0.9860820395575054 0loss/SRqqt3/detection_loss‚1.750275054978112 % _timestamp‚1.7884538972466903e+09 0loss/FRt1/assignment_loss‚2.7048795554372997 /loss/FRt3/assignment_loss‚3.005001464994942 2loss/SRqqt2/assignment_loss‚1.0783381241279235 .loss/FRt1/detection_loss‚2.435042746199502 .loss/FRt3/detection_loss‚2.421584881660415 1loss/SRqqt4/detection_loss‚2.0868641897743823 .loss/FBt3/detection_loss‚2.057752900653415 trainer/global_step‚7649 2loss/SRqqt3/assignment_loss‚1.8398245958200956 1loss/FBt1/assignment_loss‚0.00911962376825386 0loss/FBt3/assignment_loss‚0.3881659320452147 1loss/SRqqt4/assignment_loss‚1.853392414529534 1loss/FBt2/assignment_loss‚0.09086066184999585õT]\žõ‚(:“ ”ÈæÔøþóÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.759' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.551799999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž.cžHö‚(:½ ”ÈæÔ“·¡Ù disk.rbd1.out‚ 3.06640625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 190.515625 disk.nvme0n1.in‚ 190.515625( disk.nvme0c0n1.out‚26364.69873046875 disk.sda2.out‚0# disk.dm-0.out‚26364.69873046875% disk./.usageGB‚1110.2477912902832 disk.sda1.in‚0! disk.rbd2.out‚149.46337890625 network.sent‚ 1.8098054e+07) disk./.usagePercent‚74.50966083343602 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 232.53515625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0* proc.memory.percent‚17.070722579956055 cpu‚25.36587147327673 disk.sda3.out‚ 232.53515625 disk.sda2.in‚0$ memory_percent‚93.71159076690674) proc.memory.availableMB‚ 1030.29296875" proc.memory.rssMB‚ 2796.8671875 network.recv‚ 2.157417e+06 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚26364.69873046875 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.515625 disk.sda.out‚ 232.53515625¯ç“÷‚(:ˆ £ÈæÔåÒÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oêqªø‚(:· £ÈæÔ¦Û³Ø disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.070722579956055 disk.sda.out‚ 233.203125 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 190.515625# disk.dm-0.out‚26364.97216796875 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 190.515625 disk.nvme0n1.in‚ 190.515625$ memory_percent‚93.71020793914795 cpu‚25.36509460590083 network.recv‚ 2.162025e+06% disk./.usageGB‚1110.2478CzBB942871094 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17 network.sent‚ 1.8204982e+07) disk./.usagePercent‚74.50966774563943 disk.dm-1.out‚ 233.203125 disk.rbd3.out‚0 disk.sda3.out‚ 233.203125 disk.sda.in‚ 0.01171875 disk.sda1.in‚0! disk.rbd2.out‚151.03759765625( disk.nvme0c0n1.out‚26364.97216796875& disk.nvme0n1.out‚26364.97216796875 disk.sda2.in‚0 disk.rbd1.out‚ 3.06640625 disk.rbd0.in‚0) proc.memory.availableMB‚ 1030.51953125" proc.memory.rssMB‚ 2796.8671875 disk.drbd1777.in‚0Z’ö2ù- _wandbruntime‚6024  _runtime‚6024®—ÇúÁ & lr-AdamW/pg1‚6.464706976551726e-05 & lr-AdamW/pg2‚6.464706976551726e-05 trainer/global_step‚7699 % _timestamp‚1.7884539329379668e+09 _runtime‚6024.492402546 _step‚630¹3“ÌûÆ _step‚630 & lr-AdamW/pg1‚6.464706976551726e-05 & lr-AdamW/pg2‚6.464706976551726e-05 trainer/global_step‚7699 % _timestamp‚1.7884539329379668e+09 _runtime‚6024.492402546öÆ ]O2ü- _wandbruntime‚6024  _runtime‚6024‘(Qæ'ýàO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145398238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788453932938787e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚7699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112015464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145378200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454307401865728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚6024.493453026 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚631 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6449683001344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447857062501888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36833302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6426591615428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814580874240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814382824448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36833766 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454306727500288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6766044175872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145398238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788453932938787e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚7699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112015464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145378200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454307401865728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚6024.493453026 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚631 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6449683001344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447857062501888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36833302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6426591615428 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112015466 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447857044498944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814580874240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814382824448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36833766 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454306727500288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6766044175872 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102203824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36838299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6450502429184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36837844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767068141568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426753612636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877081467904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899966976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442600187530076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396654 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426134 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102204281 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚6025.179205446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102203824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436173450956800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36838299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6450502429184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36837844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767068141568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426753612636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877081467904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899966976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442600187530076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396654 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426134 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102204281 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚6025.179205446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚632 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102826402 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65766212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37060620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146304252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 259020516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6809239336960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450674696441344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 259040556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65766214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325169664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102826402 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445331180309076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450674696441344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65766212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37060620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146304252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 259020516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112716266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6809239336960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450674696441344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 259040556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65766214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6491325169664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952044 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445386229397484 _step‚636 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146322706 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102839015 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884539701952002e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810100285952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342770 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102839472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492192526269440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112730282 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚6061.750691246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144610816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144610816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6809924092416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146322706 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102839015 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884539701952002e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810100285952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146342770 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102839472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492192526269440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259073052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112730282 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚6061.750691246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144610816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492144610816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6809924092416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986ü§ª2”- _wandbruntime‚6061  _runtime‚6061»Ï ó]•× 1loss/SRqqt3/detection_loss‚1.9161316998759095 /loss/FBt1/detection_loss‚2.2519437193870546 epoch‚6 0loss/FRt3/assignment_loss‚2.7807569806392376 2loss/SRqqt1/assignment_loss‚1.4295363933482068 2loss/SRqqt2/assignment_loss‚0.9869974490946098 /loss/FRt4/detection_loss‚3.3782417376836142 /loss/FBt2/detection_loss‚0.6008367504376414 /loss/FRt2/assignment_loss‚1.666677102446556 2loss/SRqqt3/assignment_loss‚1.3040516385618521 0loss/FBt1/assignment_loss‚0.3759053074987605 0loss/FBt3/assignment_loss‚0.2500844028674894 .loss/FRt2/detection_loss‚1.890668555162847 /loss/FRt3/detection_loss‚1.9674654483795166 1loss/SRqqt4/detection_loss‚2.6945596571479524 _step‚637 0loss/FRt1/assignment_loss‚2.7875599499904746 1loss/FBt4/assignment_loss‚0.17485284976040325 1loss/SRqqt1/detection_loss‚1.6129313156484288 1loss/SRqqt2/detection_loss‚1.2514139344134638 /loss/FBt3/detection_loss‚2.3160233828756542 % _timestamp‚1.7884539701976664e+09 0loss/FBt2/assignment_loss‚0.1700611876326073 /loss/FBt4/detection_loss‚3.2621650298436484 trainer/global_step‚7749 2loss/SRqqt4/assignment_loss‚1.5426745960222823 .loss/FRt1/detection_loss‚2.803996481001377 &loss/total_loss‚43.110020066829634 /loss/FRt4/assignment_loss‚3.694484497110049 _runtime‚6061.751965949anšob–Ü 0loss/FBt2/assignment_loss‚0.1700611876326073 /loss/FBt4/detection_loss‚3.2621650298436484 trainer/global_step‚7749 2loss/SRqqt4/assignment_loss‚1.5426745960222823 .loss/FRt1/detection_loss‚2.803996481001377 &loss/total_loss‚43.110020066829634 /loss/FRt4/assignment_loss‚3.694484497110049 _runtime‚6061.751965949 1loss/SRqqt3/detection_loss‚1.9161316998759095 /loss/FBt1/detection_loss‚2.2519437193870546 epoch‚6 0loss/FRt3/assignment_loss‚2.7807569806392376 2loss/SRqqt1/assignment_loss‚1.4295363933482068 2loss/SRqqt2/assignment_loss‚0.9869974490946098 /loss/FRt4/detection_loss‚3.3782417376836142 /loss/FBt2/detection_loss‚0.6008367504376414 /loss/FRt2/assignment_loss‚1.666677102446556 2loss/SRqqt3/assignment_loss‚1.3040516385618521 0loss/FBt1/assignment_loss‚0.3759053074987605 0loss/FBt3/assignment_loss‚0.2500844028674894 .loss/FRt2/detection_loss‚1.890668555162847 /loss/FRt3/detection_loss‚1.9674654483795166 1loss/SRqqt4/detection_loss‚2.6945596571479524 _step‚637 0loss/FRt1/assignment_loss‚2.7875599499904746 1loss/FBt4/assignment_loss‚0.17485284976040325 1loss/SRqqt1/detection_loss‚1.6129313156484288 1loss/SRqqt2/detection_loss‚1.2514139344134638 /loss/FBt3/detection_loss‚2.3160233828756542 % _timestamp‚1.7884539701976664e+09ýÒjÈž—‚(:“ ßÈæÔ®ÁÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.158' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.831599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹òg‚C˜‚(:¸ ßÈæÔ€ÃÙ# memory_percent‚93.7122106552124* proc.memory.percent‚17.070722579956055 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875! disk.rbd2.out‚154.39306640625 disk.dm-0.in‚ 190.515625) proc.memory.availableMB‚ 1030.19140625% disk./.usageGB‚1109.2103729248047 disk.nvme0n1.in‚ 190.515625# disk.dm-0.out‚26628.88427734375 disk.rbd1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.8331022e+07 disk.rbd1.out‚ 3.06640625 disk.nvme0c0n1.in‚ 190.515625 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda3.out‚ 235.55859375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 235.55859375" proc.memory.rssMB‚ 2796.8671875 disk.sda.out‚ 235.55859375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚26628.88427734375& disk.nvme0n1.out‚26628.88427734375 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.361989454348 network.recv‚ 2.1811e+06) disk./.usagePercent‚74.44003881647676 disk.rbd3.in‚0 disk.sda1.in‚0Ø·çô’™‚(:‡ îÈæÔÕÇñÕ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚95.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hàHDš‚(:¹ îÈæÔ‹‚µÙ disk.sda2.in‚0 cpu‚25.361180819997465 disk.rbd1.in‚0 disk.sda1.in‚0! disk.rbd2.out‚154.49462890625( disk.nvme0c0n1.out‚26629.20068359375# memory_percent‚93.7122106552124 disk.rbd1.out‚ 3.06640625 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 236.1953125) proc.memory.availableMB‚ 1030.19140625 disk.rbd3.in‚0 disk.sda.out‚ 236.1953125& disk.nvme0n1.out‚26629.20068359375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 190.515625 disk.drbd1777.out‚0# disk.dm-0.out‚26629.20068359375 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 236.1953125 network.sent‚ 1.843977e+07% disk./.usageGB‚1109.2104377746582 disk.sda.in‚ 0.01171875" proc.memory.rssMB‚ 2796.8671875 network.recv‚ 2.187426e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 190.515625 disk.sda2.out‚0* proc.memory.percent‚17.070722579956055) disk./.usagePercent‚74.44004316860484 disk.dm-1.in‚ 0.01171875 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.515625C­\2›- _wandbruntime‚6097  _runtime‚6097y]”ÇœÁ & lr-AdamW/pg1‚6.211972928068042e-05 & lr-AdamW/pg2‚6.211972928068042e-05 trainer/global_step‚7799 % _timestamp‚1.7884540055711644e+09 _runtime‚6097.125556267 _step‚638¦äüŸÌÆ _runtime‚6097.125556267 _step‚638 & lr-AdamW/pg1‚6.211972928068042e-05 & lr-AdamW/pg2‚6.211972928068042e-05 trainer/global_step‚7799 % _timestamp‚1.7884540055711644e+09þß4÷Ù2ž- _wandbruntime‚6097  _runtime‚6097:@´ÃÓŸáO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260647368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103460824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677362176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚639 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002996736 trainer/glob— « al_step‚7799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6097.126517478 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260667408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448117222176484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884540055719244e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6531654612992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6852023145472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113417064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502065612638720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495213436355072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441608942842880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310975488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310975488 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103460395 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460024677362176 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250342 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37286210 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147230304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856488šäì' æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260647368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103460824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453492348383744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677362176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚639 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002996736 trainer/global_step‚7799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113417066 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6097.126517478 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260667408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448117222176484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884540055719244e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6531654612992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6852023145472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113417064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502065612638720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495213436355072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441608942842880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6532310975488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310975488 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103460395 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113417064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460024677362176 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250342 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37286210 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147230304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÿüúË2¡- _wandbruntime‚6097  _runtime‚6097oÐ ñ&¢ëM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853044898304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128131991040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127944440320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66182780 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884540062405257e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103473557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130416640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚7799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448172271264892 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103473104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37290324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6097.795527727’a,|º£ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884540062405257e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103473557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130416640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6508424291452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚7799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113431080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448172271264892 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103473104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37290324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6097.795527727 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147248758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_bat~JR¤<ch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853044898304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6532474040832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128131991040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502127944440320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66182780 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0€`c`Ï2¤- _wandbruntime‚6097  _runtime‚6097T²¢ÝX¥Ò 1loss/SRqqt3/assignment_loss‚1.592577590150576 0loss/FBt3/assignment_loss‚0.6572927893804652 epoch‚6 0loss/FBt4/assignment_loss‚0.4357637220527977 trainer/global_step‚7799 1loss/SRqqt1/detection_loss‚2.1190485664187593 /loss/FBt2/detection_loss‚0.5676991992087344 /loss/FRt4/detection_loss‚3.3457466860612235 2loss/SRqqt4/assignment_loss‚1.5189370159059763 0loss/SRqqt4/detection_loss‚2.392012180760503 /loss/FBt4/detection_loss‚2.4225938081741334 %loss/total_loss‚41.90593428934301 % _timestamp‚1.7884540062425382e+09 2loss/SRqqt2/assignment_loss‚0.9867102028514879 2loss/FBt1/assignment_loss‚0.012648633833224367 /loss/FRt1/detection_loss‚2.0572118629686167 -loss/FRt3/detection_loss‚2.23456845567985 0loss/SRqqt2/detection_loss‚1.063421320617199 1loss/SRqqt3/detection_loss‚1.8864751128005046 0loss/FRt1/assignment_loss‚2.7533190690103124 /loss/FRt3/assignment_loss‚2.867207329381596 1loss/SRqqt1/assignment_loss‚1.301933136332701 1loss/FBt2/assignment_loss‚0.17837259220620322 _step‚641 /loss/FRt4/assignment_loss‚3.476212011443244 /loss/FRt2/detection_loss‚1.9618560017979876 .loss/FBt1/detection_loss‚2.443983382648892 /loss/FBt3/detection_loss‚2.1914691839899336 _runtime‚6097.796833028 /loss/FRt2/assignment_loss‚1.438874435668089n"¦x]¦× /loss/FBt3/detection_loss‚2.1914691839899336 _runtime‚6097.796833028 /loss/FRt2/assignment_loss‚1.438874435668089 1loss/SRqqt3/assignment_loss‚1.592577590150576 0loss/FBt3/assignment_loss‚0.6572927893804652 epoch‚6 0loss/FBt4/assignment_loss‚0.4357637220527977 trainer/global_step‚7799 1loss/SRqqt1/detection_loss‚2.1190485664187593 /loss/FBt2/detection_loss‚0.5676991992087344 /loss/FRt4/detection_loss‚3.3457466860612235 2loss/SRqqt4/assignment_loss‚1.5189370159059763 0loss/SRqqt4/detection_loss‚2.392012180760503 /loss/FBt4/detection_loss‚2.4225938081741334 %loss/total_loss‚41.90593428934301 % _timestamp‚1.7884540062425382e+09 2loss/SRqqt2/assignment_loss‚0.9867102028514879 2loss/FBt1/assignment_loss‚0.012648633833224367 /loss/FRt1/detection_loss‚2.0572118629686167 -loss/FRt3/detection_loss‚2.23456845567985 0loss/SRqqt2/detection_loss‚1.063421320617199 1loss/SRqqt3/detection_loss‚1.8864751128005046 0loss/FRt1/assignment_loss‚2.7533190690103124 /loss/FRt3/assignment_loss‚2.867207329381596 1loss/SRqqt1/assignment_loss‚1.301933136332701 1loss/FBt2/assignment_loss‚0.17837259220620322 _step‚641 /loss/FRt4/assignment_loss‚3.476212011443244 /loss/FRt2/detection_loss‚1.9618560017979876 .loss/FBt1/detection_loss‚2.443983382648892߲瓧‚(:ˆ ýÈæÔûó­Ö gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~2Ka;¨‚(:° ýÈæÔÇÑì×$ disk.nvme0n1.out‚26629.611328125 disk.drbd1777.out‚0 disk.dm-0.in‚ 190.515625 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.dm-1.out‚ 236.5859375" proc.memory.rssMB‚ 2796.8671875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 236.5859375 disk.rbd0.out‚0 cpu‚25.3604328712637 network.recv‚ 2.191855e+06% disk./.usageGB‚1109.2104187011719 disk.nvme0n1.in‚ 190.515625 disk.sda1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.070722579956055 disk.rbd0.in‚0& disk.nvme0c0n1.out‚26629.611328125 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0) proc.memory.availableMB‚ 1030.34765625) disk./.usagePercent‚74.44004188856717 disk.nvme0c0n1.in‚ 190.515625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda.out‚ 236.5859375! disk.rbd2.out‚156.06884765625" memory_percent‚93.711256980896! disk.dm-0.out‚26629.611328125 network.sent‚ 1.844535e+07¯Æ³^ž©‚(:“ ŒÉæÔà«ÂÕ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.063' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.412599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ú#e@ª‚(:µ ŒÉæÔƒÅù× disk.drbd1777.in‚0 disk.sda2.out‚0$ memory_percent‚93.71347427368164 cpu‚25.359674349724234% disk./.usageGB‚1109.2104187011719 disk.sda2.in‚0 disk.sda3.out‚ 237.234375 disk.rbd2.in‚0( disk.nvme0c0n1.out‚26630.42919921875# disk.dm-0.out‚26630.42919921875 disk.rbd2.out‚157.7568359375 disk.nvme0c0n1.in‚ 190.515625 disk.rbd0.in‚0) disk./.usagePercent‚74.44004188856717 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875' proc.memory.availableMB‚ 1029.984375 network.sent‚ 1.8554062e+07 network.recv‚ 2.198094e+06 disk.dm-1.out‚ 237.234375 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0# proc.memory.rssMB‚ 2796.98046875) proc.memory.percent‚17.07141399383545 proc.cpu.threads‚17 disk.dm-0.in‚ 190.515625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda.out‚ 237.234375 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 190.515625& disk.nvme0n1.out‚26630.42919921875³^2«- _wandbruntime‚6133  _runtime‚6133âÅ>ǬÁ & lr-AdamW/pg2‚6.086531262954021e-05 trainer/global_step‚7849 % _timestamp‚1.7884540421476245e+09 _runtime‚6133.701999344 _step‚642 & lr-AdamW/pg1‚6.086531262954021e-05먛èÌ­Æ _runtime‚6133.701999344 _step‚642 & lr-AdamW/pg1‚6.086531262954021e-05 & lr-AdamW/pg2‚6.086531262954021e-05 trainer/global_step‚7849 % _timestamp‚1.7884540421476245e+09‚/—õÏ2®- _wandbruntime‚6133  _runtime‚6133<±À^ç'¯áO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148176394 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450902592566776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498295949298176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6895100496384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498295973238272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456310000326144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894945261056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37512229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640744960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462883315110400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104095241 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884540421485453e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444354149370880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚643 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296781312 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104095679‡î†ì'°æO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚7849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462882640744960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462883315110400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104095241 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884540421485453e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444354149370880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _step‚643 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296781312 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104095679 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148176394 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚450902592566776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498295949298176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6895100496384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114117864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498295973238272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456310000326144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6894945261056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37512229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ƒ‡Ôç“2±- _wandbruntime‚6134  _runtime‚6134{‘ęŲëM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6573459846656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6895796519936 SEDe—ëÌ[,&viceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884540428247607e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462940487590400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459846656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚505253588300800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66591182 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7849 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456366353364992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114131880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148174810 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148194874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37516711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444409070540800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6895964324864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630860 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚644 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450958313132300 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104107891 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498357600035840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487590400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37517145 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262326756 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114131882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6574116222464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148174810 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6549914068824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459846656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456366371367936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚505253588300800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66591182 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚7849 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456366353364992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114131880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148174810 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148194874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37516711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444409070540800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6895964324864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630860 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚644 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450958313132300 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104107891 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498357600035840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚462940487590400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37517145„MLL2´- _wandbruntime‚6134  _runtime‚6134kžsxYµÓ _step‚645 .loss/FRt4/assignment_loss‚3.36745505473193 1loss/FBt2/assignment_loss‚0.15010430491111149 -loss/FRt3/detection_loss‚2.29197543717566 2loss/SRqqt4/assignment_loss‚2.0344334703225355 /loss/FRt2/detection_loss‚1.7559709535802113 /loss/FBt1/detection_loss‚2.7951695919036865 epoch‚6 % _timestamp‚1.7884540428267045e+09 /loss/FRt1/detection_loss‚2.5414526000236854 0loss/SRqqt4/detection_loss‚2.109903688614185 %loss/total_loss‚43.52053970333473 /loss/FRt3/assignment_loss‚3.473707437515259 1loss/FBt1/assignment_loss‚0.31657002877825235 1loss/FBt3/assignment_loss‚0.08125245136519273 _runtime‚6134.381010821 2loss/SRqqt1/assignment_loss‚1.3523751310023822 0loss/SRqqt2/detection_loss‚1.079213373363018 /loss/FBt3/detection_loss‚2.6180806855360665 /loss/FBt4/detection_loss‚2.9656513002183704 2loss/SRqqt3/assignment_loss‚2.0038030321896074 /loss/FBt2/detection_loss‚0.5412280715009059 trainer/global_step‚7849 /loss/FRt1/assignment_loss‚3.376142130448268 2loss/SRqqt2/assignment_loss‚1.1952445070300974 1loss/FBt4/assignment_loss‚0.20640187594108206 /loss/FRt4/detection_loss‚2.7941790643860314 /loss/SRqqt1/detection_loss‚1.62698868466051 0loss/FRt2/assignment_loss‚1.2126117652537776 1loss/SRqqt3/detection_loss‚1.6306250628829002Ͷ^¶Ø /loss/FRt3/assignment_loss‚3.473707437515259 1loss/FBt1/assignment_loss‚0.31657002877825235 1loss/FBt3/assignment_loss‚0.08125245136519273 _runtime‚6134.381010821 2loss/SRqqt1/assignment_loss‚1.3523751310023822 0loss/SRqqt2/detection_loss‚1.079213373363018 /loss/FBt3/detection_loss‚2.6180806855360665 /loss/FBt4/detection_loss‚2.9656513002183704 2loss/SRqqt3/assignment_loss‚2.0038030321896074 /loss/FBt2/detection_loss‚0.5412280715009059 trainer/global_step‚7849 /loss/FRt1/assignment_loss‚3.376142130448268 2loss/SRqqt2/assignment_loss‚1.1952445070300974 1loss/FBt4/assignment_loss‚0.20640187594108206 /loss/FRt4/detection_loss‚2.7941790643860314 /loss/SRqqt1/detection_loss‚1.62698868466051 0loss/FRt2/assignment_loss‚1.2126117652537776 1loss/SRqqt3/detection_loss‚1.6306250628829002 _step‚645 .loss/FRt4/assignment_loss‚3.36745505473193 1loss/FBt2/assignment_loss‚0.15010430491111149 -loss/FRt3/detection_loss‚2.29197543717566 2loss/SRqqt4/assignment_loss‚2.0344334703225355 /loss/FRt2/detection_loss‚1.7559709535802113 /loss/FBt1/detection_loss‚2.7951695919036865 epoch‚6 % _timestamp‚1.7884540428267045e+09 /loss/FRt1/detection_loss‚2.5414526000236854 0loss/SRqqt4/detection_loss‚2.109903688614185 %loss/total_loss‚43.52053970333473…"'P“·‚(:ˆ ›ÉæÔ¼þÖ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.351' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ)WG¸‚(:¼ ›ÉæÔýʤØ network.recv‚ 2.201923e+06% disk./.usageGB‚1109.2103805541992 disk.rbd3.in‚0 disk.sda.out‚ 237.62890625 disk.rbd0.in‚0) proc.memory.percent‚17.07141399383545 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 190.515625& disk.nvme0n1.out‚26630.73388671875 disk.drbd1777.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2796.98046875 cpu‚25.35891741903973 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0 disk.rbd2.out‚157.7568359375 disk.sda2.out‚0) disk./.usagePercent‚74.44003932849182 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8559369e+07# disk.dm-0.out‚26630.73388671875 disk.dm-1.in‚ 0.01171875$ memory_percent‚93.71187686920166( disk.nvme0c0n1.out‚26630.73388671875 disk.drbd1777.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 237.62890625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1030.24609375 disk.nvme0n1.in‚ 190.515625 disk.dm-0.in‚ 190.515625 disk.dm-1.out‚ 237.62890625 disk.sda3.in‚ 0.01171875d;Ož¹‚(:“ ªÉæÔ›âîÖ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.697000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ÃˆCº‚(:¸ ªÉæÔÚÉØ disk.sda1.in‚0 disk.sda3.out‚ 238.1171875 disk.sda2.out‚0 disk.sda.in‚ 0.01171875$ memory_percent‚93.71213912963867) proc.memory.percent‚17.07141399383545 disk.nvme0c0n1.in‚ 190.515625# disk.dm-0.out‚26631.15380859375 disk.rbd3.in‚0% disk./.usageGB‚1109.2102012634277 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 238.1171875) disk./.usagePercent‚74.44002729613773( disk.nvme0c0n1.out‚26631.15380859375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 238.1171875 proc.cpu.threads‚17 network.sent‚ 1.8668087e+07& disk.nvme0n1.out‚26631.15380859375 disk.dm-0.in‚ 190.515625 disk.rbd1.out‚ 3.06640625' proc.memory.availableMB‚ 1030.203125 disk.nvme0n1.in‚ 190.515625 disk.rbd1.in‚0 network.recv‚ 2.208143e+06 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.out‚159.4951171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2796.98046875 disk.rbd2.in‚0 cpu‚25.358164736318628ÙL¡ž»‚(:“ ¹ÉæÔùŸïÖ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.063' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.412599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë{W¦G¼‚(:¼ ¹ÉæÔ²ÒýØ disk.rbd2.out‚159.4951171875 disk.sda2.in‚0) proc.memory.availableMB‚ 1030.19921875# proc.memory.rssMB‚ 2796.98046875 network.recv‚ 2.212512e+06 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 238.92578125 disk.rbd3.out‚0) proc.memory.percent‚17.07141399383545 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.06640625 cpu‚25.357414490319773 network.sent‚ 1.867368e+07 disk.sda3.out‚ 238.92578125# disk.dm-0.out‚26631.44677734375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 238.92578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 190.515625( disk.nvme0c0n1.out‚26631.44677734375& disk.nvme0n1.out‚26631.44677734375 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.515625% disk./.usageGB‚1109.2102394104004 disk.sda1.out‚0 disk.nvme0n1.in‚ 190.515625 disk.rbd1.in‚0$ memory_percent‚93.71216297149658) disk./.usagePercent‚74.44002985621306 disk.sda3.in‚ 0.01171875fGZ2½- _wandbruntime‚6169  _runtime‚6169툹ßǾÁ & lr-AdamW/pg1‚5.961741403194317e-05 & lr-AdamW/pg2‚5.961741403194317e-05 trainer/global_step‚7899 % _timestamp‚1.7884540779360619e+09 _runtime‚6169.490501891 _step‚646©«püÌ¿Æ _step‚646 & lr-AdamW/pg1‚5.961741403194317e-05 & lr-AdamW/pg2‚5.961741403194317e-05 trainer/global_step‚7899 % _timestamp‚1.7884540779360619e+09 _runtime‚6169.490501891†ëÖY2À- _wandbruntime‚6169  _runtime‚6169|º¿îÁáO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937865868800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6589932973060 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚6169.491452172 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149082408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459127652268544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263901072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104729847 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508316558488576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114818664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614282587136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459127670271488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626224640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884540779368162e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447099372938240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6589278535304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66991414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37737966 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6938048367104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501378486181376 \HDeviceStatsMo51`ùnitor.on_train_batch_start/requested_bytes.large_pool.freed‚447099355898880 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114818666 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952858624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104729378 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453689305911300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278493184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453688634434184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459127652268544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚459127670271488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282587136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263921112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508316352050176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚501378510121472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114818666 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6613626224640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114818666 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206438400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6613626224640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚6169.491452172 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149102446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149082408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚459127652268544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263901072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182498304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104729847 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚508316558488576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114818664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614282587136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459127670271488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626224640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884540779368162e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447099372938240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6589278535304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66991414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37737966 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6938048367104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114818664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501378486181376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447099355898880 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114818666 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465741952858624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104729378 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453689305911300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278493184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453688634434184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496‡î{î2Ã- _wandbruntime‚6170  _runtime‚6170ß«§2ñ&ÄëM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37742459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884540786186473e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104742039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938885908480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102028288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100862 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114832680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37742920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379046767616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104742502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445652480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938709714944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459184005307392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6614445652480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430zÄÕÅðM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445652480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938709714944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459184005307392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6614445652480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149120926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37742459 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501440160859136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66999580 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884540786186473e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104742039 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938885908480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102028288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114832682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149©sèÀ! 100862 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114832680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚447154260029440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37742920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379046767616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104742502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066ˆrâEË2Æ- _wandbruntime‚6170  _runtime‚6170þŠä[[ÇÕ /loss/FBt1/detection_loss‚2.9526185989379883 1loss/SRqqt3/detection_loss‚1.8064794721556645 .loss/FBt4/detection_loss‚3.136573572953542 2loss/SRqqt4/assignment_loss‚1.4437479984891766 0loss/FBt3/assignment_loss‚0.5348694640426681 /loss/FRt3/detection_loss‚2.2414127922639624 .loss/FRt4/detection_loss‚3.362100576812571 /loss/FRt1/detection_loss‚2.2592826846034026 1loss/SRqqt1/detection_loss‚1.6119702257986726 1loss/SRqqt2/detection_loss‚1.0187665771015666 -loss/FBt3/detection_loss‚2.09830156656412 epoch‚6 _step‚649 0loss/FRt2/assignment_loss‚1.2503060947765003 /loss/FRt2/detection_loss‚1.8497556334192102 1loss/SRqqt4/detection_loss‚2.4277010402258705 % _timestamp‚1.7884540786207051e+09 0loss/FRt1/assignment_loss‚3.0112600243368814 2loss/SRqqt1/assignment_loss‚1.4627355803928241 2loss/SRqqt2/assignment_loss‚0.9365621861634832 1loss/FBt2/assignment_loss‚0.23242875886580733 _runtime‚6170.174995149 0loss/FRt3/assignment_loss‚3.8571348466524262 1loss/FBt1/assignment_loss‚0.20665976051241158 /loss/FBt2/detection_loss‚0.4662435499470847 %loss/total_loss‚44.17411053086059 .loss/FRt4/assignment_loss‚3.93304457989606 2loss/SRqqt3/assignment_loss‚1.2628309210652815 trainer/global_step‚7899 0loss/FBt4/assignment_loss‚0.8113240248834094^¢¸p`ÈÚ 2loss/SRqqt1/assignment_loss‚1.4627355803928241 2loss/SRqqt2/assignment_loss‚0.9365621861634832 1loss/FBt2/assignment_loss‚0.23242875886580733 _runtime‚6170.174995149 0loss/FRt3/assignment_loss‚3.8571348466524262 1loss/FBt1/assignment_loss‚0.20665976051241158 /loss/FBt2/detection_loss‚0.4662435499470847 %loss/total_loss‚44.17411053086059 .loss/FRt4/assignment_loss‚3.93304457989606 2loss/SRqqt3/assignment_loss‚1.2628309210652815 trainer/global_step‚7899 0loss/FBt4/assignment_loss‚0.8113240248834094 /loss/FBt1/detection_loss‚2.9526185989379883 1loss/SRqqt3/detection_loss‚1.8064794721556645 .loss/FBt4/detection_loss‚3.136573572953542 2loss/SRqqt4/assignment_loss‚1.4437479984891766 0loss/FBt3/assignment_loss‚0.5348694640426681 /loss/FRt3/detection_loss‚2.2414127922639624 .loss/FRt4/detection_loss‚3.362100576812571 /loss/FRt1/detection_loss‚2.2592826846034026 1loss/SRqqt1/detection_loss‚1.6119702257986726 1loss/SRqqt2/detection_loss‚1.0187665771015666 -loss/FBt3/detection_loss‚2.09830156656412 epoch‚6 _step‚649 0loss/FRt2/assignment_loss‚1.2503060947765003 /loss/FRt2/detection_loss‚1.8497556334192102 1loss/SRqqt4/detection_loss‚2.4277010402258705 % _timestamp‚1.7884540786207051e+09 0loss/FRt1/assignment_loss‚3.0112600243368814‰W{ÿўɂ(:“ ÈÉæÔ€ÉúÖ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚136.947' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.389400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ßnÑHÊ‚(:½ ÈÉæÔïÊÚÙ( disk.nvme0c0n1.out‚26633.77783203125 disk.rbd1.out‚ 3.06640625) proc.memory.percent‚17.07141399383545& disk.nvme0n1.out‚26633.77783203125 disk.dm-0.in‚ 190.515625 disk.dm-1.out‚ 239.58984375 disk.sda.out‚ 239.58984375 proc.cpu.threads‚17 disk.rbd2.out‚161.1748046875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 239.58984375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.8680027e+07) disk./.usagePercent‚74.42875477239005% disk./.usageGB‚1109.0422325134277 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875# proc.memory.rssMB‚ 2796.98046875 cpu‚25.356665286827507 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 190.515625 disk.drbd1777.out‚0# disk.dm-0.out‚26633.77783203125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 190.515625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875$ memory_percent‚93.71192455291748) proc.memory.availableMB‚ 1030.23828125 network.recv‚ 2.217026e+06J7U\’Ë‚(:‡ ×ÉæÔª›ûÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚92.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̤u ;Ì‚(:° ×ÉæÔšô¶Ø disk.rbd1.in‚0 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2796.98046875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 190.515625 disk.dm-0.in‚ 190.515625$ memory_percent‚93.71261596679688 cpu‚25.355976786290487 disk.rbd3.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda.out‚ 240.2890625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚26634.9599609375 disk.dm-1.out‚ 240.2890625 proc.cpu.threads‚17) disk./.usagePercent‚74.44462493544513 disk.sda1.out‚0 disk.rbd2.out‚161.1748046875 disk.sda2.in‚0) proc.memory.percent‚17.07141399383545 network.sent‚ 1.8787847e+07 network.recv‚ 2.22241e+06 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 190.515625% disk.nvme0n1.out‚26634.9599609375 disk.drbd1777.out‚0$ proc.memory.availableMB‚1030.125 disk.rbd3.in‚0 disk.drbd1777.in‚0" disk.dm-0.out‚26634.9599609375 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875$ disk./.usageGB‚1109.278709411621 disk.sda3.out‚ 240.2890625 disk.rbd0.out‚0~ÃãP2Í- _wandbruntime‚6206  _runtime‚6206à4û9ÉÎà ' lr-AdamW/pg1‚5.8376293548086634e-05 ' lr-AdamW/pg2‚5.8376293548086634e-05 trainer/global_step‚7949 % _timestamp‚1.7884541144543498e+09 _runtime‚6206.008751331 _step‚650YnbpÎÏÈ _runtime‚6206.008751331 _step‚650 ' lr-AdamW/pg1‚5.8376293548086634e-05 ' lr-AdamW/pg2‚5.8376293548086634e-05 trainer/global_step‚7949 % _timestamp‚1.7884541144543498e+09Šë¶2Ð- _wandbruntime‚6206  _runtime‚6206*ç'ÑáO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461945322213888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6980953388032 trainer/global_step‚7949 % _timestamp‚1.7884541144550564e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105363661 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980785569792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590606848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511441808634368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461945304210944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461945322213888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612030464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268392960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚6206.009607384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚651 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028498 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456475347778708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008460Lhkì'ÒæO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461945322213888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612030464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268392960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚6206.009607384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚651 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028498 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456475347778708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6630113874712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461945322213888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6980953388032 trainer/global_step‚7949 % _timestamp‚1.7884541144550564e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105363661 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6980785569792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590606848 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511441808634368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67399812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚461945304210944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0‹õœ‰ü2Ó- _wandbruntime‚6206  _runtime‚6206hÄlÏ ÔëM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456531068344440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67407982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462001657249792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37968376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468657088708096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 99555860ŒÏ¡"48 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511504333768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150026914 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105376356 % _timestamp‚1.7884541151255038e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468657763086848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚6206.680576356 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115533480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6656087834112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504515027456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚7949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37968811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6631584747640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981659965952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ñ?ö&ÕðM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37968376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468657088708096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511504333768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150026914 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105376356 % _timestamp‚1.7884541151255038e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468657763086848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚6206.680576356 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115533480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6656087834112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511504515027456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚7949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462001657249792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37968811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6631584747640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6981659965952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456531068344440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67407982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462001657249792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414Œi"#2Ö- _wandbruntime‚6206  _runtime‚6206ñEVÝ]×× 1loss/SRqqt4/detection_loss‚2.2612463509714282 .loss/FBt2/detection_loss‚0.456520174579587 0loss/FRt3/assignment_loss‚2.7094423579719833 1loss/SRqqt3/assignment_loss‚1.330501005429317 1loss/FBt2/assignment_loss‚0.16557293342709214 1loss/SRqqt3/detection_loss‚1.6856919864290638 _runtime‚6206.681919557 /loss/FRt4/assignment_loss‚4.088200976771693 /loss/FBt4/detection_loss‚2.2266443848609923 _step‚653 2loss/SRqqt1/assignment_loss‚1.5335505452866738 1loss/SRqqt2/assignment_loss‚1.133203507307917 1loss/FBt1/assignment_loss‚0.08050404514651746 1loss/FBt3/assignment_loss‚0.05510824563680217 /loss/FBt3/detection_loss‚3.3996448516845703 epoch‚6 % _timestamp‚1.7884541151276345e+09 /loss/FBt1/detection_loss‚2.6205043733119964 /loss/FRt1/detection_loss‚2.2276396085413133 .loss/FRt2/detection_loss‚1.713450572681877 0loss/FRt2/assignment_loss‚1.4391756979924328 1loss/FBt4/assignment_loss‚0.25384205477312205 &loss/total_loss‚41.380694482486675 0loss/FRt1/assignment_loss‚2.4140543365965086 2loss/SRqqt4/assignment_loss‚1.6675235863674331 /loss/FRt3/detection_loss‚2.0422742102506026 .loss/FRt4/detection_loss‚2.782579320092355 0loss/SRqqt1/detection_loss‚1.953990765202504 1loss/SRqqt2/detection_loss‚1.1398285911728938 trainer/global_step‚7949–sŠìbØÜ /loss/FBt1/detection_loss‚2.6205043733119964 /loss/FRt1/detection_loss‚2.2276396085413133 .loss/FRt2/detection_loss‚1.713450572681877 0loss/FRt2/assignment_loss‚1.4391756979924328 1loss/FBt4/assignment_loss‚0.25384205477312205 &loss/total_loss‚41.380694482486675 0loss/FRt1/assignment_loss‚2.4140543365965086 2loss/SRqqt4/assignment_loss‚1.6675235863674331 /loss/FRt3/detection_loss‚2.0422742102506026 .loss/FRt4/detection_loss‚2.782579320092355 0loss/SRqqt1/detection_loss‚1.953990765202504 1loss/SRqqt2/detection_loss‚1.1398285911728938 trainer/global_step‚7949 1loss/SRqqt4/detection_loss‚2.2612463509714282 .loss/FBt2/detection_loss‚0.456520174579587 0loss/FRt3/assignment_loss‚2.7094423579719833 1loss/SRqqt3/assignment_loss‚1.330501005429317 1loss/FBt2/assignment_loss‚0.16557293342709214 1loss/SRqqt3/detection_loss‚1.6856919864290638 _runtime‚6206.681919557 /loss/FRt4/assignment_loss‚4.088200976771693 /loss/FBt4/detection_loss‚2.2266443848609923 _step‚653 2loss/SRqqt1/assignment_loss‚1.5335505452866738 1loss/SRqqt2/assignment_loss‚1.133203507307917 1loss/FBt1/assignment_loss‚0.08050404514651746 1loss/FBt3/assignment_loss‚0.05510824563680217 /loss/FBt3/detection_loss‚3.3996448516845703 epoch‚6 % _timestamp‚1.7884541151276345e+09[ÞÉžÙ‚(:“ æÉæÔèöØÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.758' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.151600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂxFNÚ‚(:à æÉæÔ㟑Ø disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚162.9130859375 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.51953125) proc.memory.percent‚17.07141399383545 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625 disk.sda.out‚ 240.71484375" disk.nvme0c0n1.in‚ 190.51953125& disk.nvme0n1.out‚26764.20458984375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 240.71484375 disk.nvme0n1.in‚ 190.51953125 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 240.71484375) proc.memory.availableMB‚ 1030.14453125 cpu‚25.355203150127146) disk./.usagePercent‚74.44550022520471 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚26764.20458984375# proc.memory.rssMB‚ 2796.98046875% disk./.usageGB‚1109.2917518615723 network.sent‚ 1.8793991e+07 network.recv‚ 2.226767e+06 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚93.71249675750732# disk.dm-0.out‚26764.20458984375 disk.sda2.out‚0q{&}Û‚(:’ õÉæÔû‰‹Ö gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚96.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.293599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úõÂiJÜ‚(:¿ õÉæÔÞì¾Ø disk.rbd3.in‚0 disk.dm-0.in‚ 190.51953125 cpu‚25.354467786481% disk./.usageGB‚1109.2914581298828 disk.sda1.in‚0( disk.nvme0c0n1.out‚26891.24853515625 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 1030.3515625# proc.memory.rssMB‚ 2796.98046875 disk.rbd3.out‚0 disk.sda.out‚ 241.28515625 disk.rbd2.out‚164.4873046875 disk.sda2.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda3.out‚ 241.28515625 disk.rbd0.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 190.51953125 disk.drbd1777.out‚0# disk.dm-0.out‚26891.24853515625 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.8902099e+07) disk./.usagePercent‚74.44548051262457 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚26891.24853515625 disk.dm-1.out‚ 241.28515625 disk.rbd1.in‚0) proc.memory.percent‚17.07141399383545 network.recv‚ 2.232568e+06 disk.rbd2.in‚0 disk.nvme0n1.in‚ 190.51953125 disk.dm-1.in‚ 0.01171875$ memory_percent‚93.71123313903809ðûZ“Ý‚(:ˆ „ÊæÔàËéÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“'@Þ‚(:µ „ÊæÔ­¤³Ù# proc.memory.rssMB‚ 2796.98046875 disk.sda3.out‚ 241.90625 disk.sda.out‚ 241.90625 disk.sda1.out‚0 network.recv‚ 2.23717e+06 disk.rbd2.out‚164.6552734375" disk.nvme0c0n1.in‚ 190.51953125 disk.sda3.in‚ 0.01171875% disk./.usageGB‚1109.2915954589844 disk.rbd3.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚26892.0615234375" disk.dm-0.out‚26892.0615234375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 190.51953125 disk.dm-1.out‚ 241.90625) proc.memory.percent‚17.07141399383545 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 190.51953125 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.35377113867175 proc.cpu.threads‚17 network.sent‚ 1.8908461e+07' disk.nvme0c0n1.out‚26892.0615234375 disk.sda2.out‚0) disk./.usagePercent‚74.44548972889581 disk.sda.in‚ 0.01171875$ memory_percent‚93.71569156646729) proc.memory.availableMB‚ 1029.62109375{‘È_2ß- _wandbruntime‚6244  _runtime‚6244K¶50Éàà ' lr-AdamW/pg1‚5.7142209825619224e-05 ' lr-AdamW/pg2‚5.7142209825619224e-05 trainer/global_step‚7999 % _timestamp‚1.7884541526984346e+09 _runtime‚6244.252814107 _step‚654(röÎáÈ ' lr-AdamW/pg1‚5.7142209825619224e-05 ' lr-AdamW/pg2‚5.7142209825619224e-05 trainer/global_step‚7999 % _timestamp‚1.7884541526984346e+09 _runtime‚6244.252814107 _step‚654Ž?5S2â- _wandbruntime‚6244  _runtime‚6244¤Ù=4æãáO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150934512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954550 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471459228355072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464762956153344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67808212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6696254198784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105998624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884541526991608e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459260718169000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚6244.253821054 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023927045120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7023761324032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116220266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514567510932992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚7999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954550 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @£vt8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597836288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6695597836288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116220264 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105998167 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67808212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6696254198784 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105998624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884541526991608e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459260718169000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚6244.253821054 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7023927045120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7023761324032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116220266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514567510932992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚7999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954550 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597836288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6695597836288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116220264 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105998167 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507543559947776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514629827317248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087048 trainer/global_step‚7999 _runtime‚6244.923834544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884541533689237e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6696417264128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973030 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106010878 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38194948 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952966 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400835072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452644690124800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459317110211848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464819327195136 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459317110211848 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464819327195136 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116234282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514629827317248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6672420087048 trainer/global_step‚7999 _runtime‚6244.923834544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884541533689237e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6696417264128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973030 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106010878 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38194948 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952966 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400835072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464819309192192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452644690124800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507605210685440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692812341248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38417452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7066883328512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6736583642112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268801668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116921064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880602 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884541890795753e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38417900 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106634514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 trainer/global_step‚8049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517692812341248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38417452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7066883328512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6736583642112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268801668 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116921064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880602 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884541890795753e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38417900 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106634514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 trainer/global_step‚8049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467636961134592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106646778 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517755521486848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38421998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6738059445760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467636979137536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116935082 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚6281.330770935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6738059445760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038583296 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517755348616192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462102480601932 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38422444 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151899082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7067749978112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116935080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106647226 H@DeviceStatsMonitor.on_*™4train_batch_end/inactive_split.all.current‚448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6713255426456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7067601047552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467636961134592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455389879613440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884541897757554e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467636979137536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988492 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106647226 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6713255426456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7067601047552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467636961134592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68224782 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455389879613440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884541897757554e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467636979137536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988492 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚660 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467636961134592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106646778 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517755521486848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38421998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6738059445760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467636979137536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116935082 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚6281.330770935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6738059445760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474375038583296 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517755348616192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462102480601932 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38422444 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430”jˆòo2ø- _wandbruntime‚6281  _runtime‚6281þ.Q¯VùÐ 0loss/FRt3/assignment_loss‚3.3550085619438526 0loss/FRt1/assignment_loss‚2.5966973875431303 /loss/FRt4/assignment_loss‚4.156456213444471 /loss/FBt2/detection_loss‚0.6658666884067698 2loss/SRqqt4/assignment_loss‚1.8209393624748502 1loss/SRqqt1/detection_loss‚1.7041827172546062 .loss/FBt1/detection_loss‚2.195639397416796 _runtime‚6281.332431384 2loss/SRqqt1/assignment_loss‚1.4837913878135762 2loss/SRqqt2/assignment_loss‚1.1296058776379938 1loss/SRqqt3/assignment_loss‚1.385262674518994 1loss/FBt2/assignment_loss‚0.19355506665013608 /loss/FRt1/detection_loss‚2.1887903996604554 /loss/FRt2/detection_loss‚1.5028236813710205 .loss/FRt3/detection_loss‚2.201410716356233 /loss/FRt4/detection_loss‚2.7652654871344566 0loss/SRqqt4/detection_loss‚2.506293739591326 /loss/FBt4/detection_loss‚3.9351144234339395 %loss/total_loss‚44.48122664456886 epoch‚6 trainer/global_step‚8049 /loss/FBt3/assignment_loss‚0.572529148356989 0loss/FBt4/assignment_loss‚0.9840136567751566 % _timestamp‚1.7884541897780604e+09 1loss/SRqqt2/detection_loss‚1.0965705843348252 /loss/FRt2/assignment_loss‚1.456955428634371 /loss/FBt1/assignment_loss‚0.646846231838156 /loss/SRqqt3/detection_loss‚1.72486573963293 /loss/FBt3/detection_loss‚2.2127420723438265 _step‚661¡I¬[úÕ /loss/FBt3/assignment_loss‚0.572529148356989 0loss/FBt4/assignment_loss‚0.9840136567751566 % _timestamp‚1.7884541897780604e+09 1loss/SRqqt2/detection_loss‚1.0965705843348252 /loss/FRt2/assignment_loss‚1.456955428634371 /loss/FBt1/assignment_loss‚0.646846231838156 /loss/SRqqt3/detection_loss‚1.72486573963293 /loss/FBt3/detection_loss‚2.2127420723438265 _step‚661 0loss/FRt3/assignment_loss‚3.3550085619438526 0loss/FRt1/assignment_loss‚2.5966973875431303 /loss/FRt4/assignment_loss‚4.156456213444471 /loss/FBt2/detection_loss‚0.6658666884067698 2loss/SRqqt4/assignment_loss‚1.8209393624748502 1loss/SRqqt1/detection_loss‚1.7041827172546062 .loss/FBt1/detection_loss‚2.195639397416796 _runtime‚6281.332431384 2loss/SRqqt1/assignment_loss‚1.4837913878135762 2loss/SRqqt2/assignment_loss‚1.1296058776379938 1loss/SRqqt3/assignment_loss‚1.385262674518994 1loss/FBt2/assignment_loss‚0.19355506665013608 /loss/FRt1/detection_loss‚2.1887903996604554 /loss/FRt2/detection_loss‚1.5028236813710205 .loss/FRt3/detection_loss‚2.201410716356233 /loss/FRt4/detection_loss‚2.7652654871344566 0loss/SRqqt4/detection_loss‚2.506293739591326 /loss/FBt4/detection_loss‚3.9351144234339395 %loss/total_loss‚44.48122664456886 epoch‚6 trainer/global_step‚8049•´=ž‘û‚(:† ±ÊæÔîãöÕ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X<ÙûCü‚(:¸ ±ÊæÔêêÜØ$ disk./.usageGB‚1109.300868988037 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 243.7265625 disk.rbd2.out‚167.9912109375 disk.nvme0n1.in‚ 190.51953125 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.51953125$ memory_percent‚93.71681213378906" disk.dm-0.out‚26901.2509765625 network.recv‚ 2.251892e+06 disk.sda.in‚ 0.01171875 disk.sda1.out‚0' disk.nvme0c0n1.out‚26901.2509765625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.06640625% proc.memory.availableMB‚ 1029.4375" disk.nvme0c0n1.in‚ 190.51953125 disk.rbd1.in‚0 cpu‚25.351571049288445 proc.cpu.threads‚17 network.sent‚ 1.9027949e+07) disk./.usagePercent‚74.44611208321155 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0% disk.nvme0n1.out‚26901.2509765625 disk.sda2.out‚0 disk.dm-1.out‚ 243.7265625# proc.memory.rssMB‚ 2796.98046875 disk.rbd3.out‚0 disk.sda.out‚ 243.7265625 disk.sda2.in‚0) proc.memory.percent‚17.07141399383545Óøåb‘ý‚(:† ÀÊæÔ¬Ø¬Õ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚163.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ŒÁèMþ‚(: ÀÊæÔ×Å€Ø proc.cpu.threads‚17( disk.nvme0c0n1.out‚26901.74365234375& disk.nvme0n1.out‚26901.74365234375 cpu‚25.350861926905356 disk.dm-1.out‚ 244.23046875 disk.sda1.in‚0 disk.sda3.out‚ 244.23046875 disk.rbd0.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1029.28515625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 1.9136933e+07 disk.drbd1777.out‚0$ memory_percent‚93.71774196624756 network.recv‚ 2.25853e+06 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚74.44611643533963 disk.rbd1.out‚ 3.06640625 disk.rbd2.out‚168.1708984375" disk.nvme0c0n1.in‚ 190.51953125 disk.nvme0n1.in‚ 190.51953125) proc.memory.percent‚17.07141399383545 disk.sda1.out‚0% disk./.usageGB‚1109.3009338378906 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 244.23046875# proc.memory.rssMB‚ 2796.98046875 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.51953125# disk.dm-0.out‚26901.743652343751å6žÿ‚(:“ ÏÊæÔÀ½ÈÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.623' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.724599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G?GF€ ‚(:» ÏÊæÔÕÏÙ disk.sda3.out‚ 244.6953125% disk./.usageGB‚1109.3009757995605 disk.drbd1777.in‚0" disk.dm-0.out‚26902.0927734375 disk.dm-1.out‚ 244.6953125 disk.rbd1.in‚0) proc.memory.availableMB‚ 1029.40234375) proc.memory.percent‚17.07141399383545% disk.nvme0n1.out‚26902.0927734375 disk.dm-0.in‚ 190.51953125 disk.rbd1.out‚ 3.06640625 disk.rbd3.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 190.51953125 disk.sda2.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2796.98046875 cpu‚25.35013682895045 network.recv‚ 2.262339e+06' disk.nvme0c0n1.out‚26902.0927734375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚169.7451171875 disk.nvme0n1.in‚ 190.51953125 disk.sda.out‚ 244.6953125$ memory_percent‚93.71702671051025 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 network.sent‚ 1.9142291e+07( disk./.usagePercent‚74.4461192514225 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875ƒz^2 - _wandbruntime‚6316  _runtime‚6316£\ábÇ‚ Á & lr-AdamW/pg1‚5.469617986959732e-05 & lr-AdamW/pg2‚5.469617986959732e-05 trainer/global_step‚8099 % _timestamp‚1.7884542250257287e+09 _runtime‚6316.580120803 _step‚662Sõ Æ % _timestamp‚1.7884542250257287e+09 _runtime‚6316.580120803 _step‚662 & lr-AdamW/pg1‚5.469617986959732e-05 & lr-AdamW/pg2‚5.469617986959732e-05 trainer/global_step‚8099–ãõÑ2„ - _wandbruntime‚6316  _runtime‚6316ôŽªÙç'… áO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38645275 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109807775232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68625012 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚663 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152786616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6778225810432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152786616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6753274330692 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117621866 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6778225810432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38644823 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470398278041088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107270289 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6752619892936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806654 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477176503851520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107269835 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117621866 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513708657654272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117621864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884542250265553e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7109971399168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470398260038144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38645275 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109807775232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68625012 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚663 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152786616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6778225810432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152786616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6753274330692 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117621866 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6778225810432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38644823 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470398278041088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107270289 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6752619892936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806654 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477176503851520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107269835 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117621866 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513708657654272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117621864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884542250265553e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7109971399168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470398260038144 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470454613076992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107282560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520880963243008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚6317.262989298 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513770308391936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7110834013184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884542257079782e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388875776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107283011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881142405120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464888522469340 _step‚664 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805070 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68633180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38649829 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560¤T¥øö&‰ ðM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513770284451840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107282560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520880963243008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚6317.262989298 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513770308391936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7110834013184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884542257079782e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388875776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107283011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881142405120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464888522469340 _step‚664 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765864 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805070 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68633180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38649829 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470454613076992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B`¾Ì“ Æ & lr-AdamW/pg1‚5.348474338501948e-05 & lr-AdamW/pg2‚5.348474338501948e-05 trainer/global_step‚8149 % _timestamp‚1.7884542609865623e+09 _runtime‚6352.540955092 _step‚666šÑ_v2” - _wandbruntime‚6352  _runtime‚6352J« ¹ç'• áO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107906845 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141599744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473215929983488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473215929983488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943885282304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460825388538880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107906389 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚6352.541916754 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6818555253760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467618843771224 trainer/global_step‚8149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272055372 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480034467234304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118322664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153712668 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69033412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚667 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670100 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521ç…ÏÜì'– æO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153712668 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884542609872625e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523944074943488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153732706 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152714684928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚6352.541916754 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6818555253760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467618843771224 trainer/global_step‚8149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272055372 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480034467234304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118322664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153712668 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69033412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚667 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670100 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107906845 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480035141599744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473215929983488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚473215929983488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523943885282304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460825388538880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69033414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107906389 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107919156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460880309708800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38878046 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153731813376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6353.211605413 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153557716992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031057408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314079744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚8149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107919628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚460880292669440 _step‚668 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 19803648W›Æ00 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639700992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470PãÀ?ö&™ ðM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107919156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460880309708800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38878046 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153731813376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚473272265019392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6353.211605413 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7153557716992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031057408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314079744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚8149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107919628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚460880292669440 _step‚668 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480091639700992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872œ»YêP2š - _wandbruntime‚6353  _runtime‚6353©iÛ^› Ø 2loss/SRqqt1/assignment_loss‚1.4930928233404492 .loss/FRt3/detection_loss‚2.073513918647579 1loss/SRqqt1/detection_loss‚1.8910680096163306 /loss/FBt4/detection_loss‚2.7350150412983365 0loss/FRt3/assignment_loss‚3.1978373118475374 1loss/FBt4/assignment_loss‚0.14245404402449444 .loss/FBt3/detection_loss‚4.570625305175781 2loss/SRqqt2/assignment_loss‚0.8187664287656339 2loss/SRqqt3/assignment_loss‚1.3725946648961067 1loss/FBt3/assignment_loss‚0.21068098465889307 .loss/FRt4/detection_loss‚2.796214919537306 1loss/SRqqt2/detection_loss‚0.8928631377509899 .loss/FBt1/detection_loss‚2.857931411266327 /loss/FRt4/assignment_loss‚3.891396563500166 % _timestamp‚1.7884542616583607e+09 0loss/FRt1/assignment_loss‚2.9274124474060246 /loss/FRt2/detection_loss‚1.4839726212349804 1loss/SRqqt3/detection_loss‚1.6389493042869228 0loss/SRqqt4/detection_loss‚2.430712789297104 0loss/FRt2/assignment_loss‚0.9722513697364114 2loss/SRqqt4/assignment_loss‚1.7500951886177063 2loss/FBt1/assignment_loss‚0.016671459882581986 /loss/FBt2/detection_loss‚0.7486499840734506 &loss/total_loss‚43.535134342405485 0loss/FBt2/assignment_loss‚0.1851929173815751 /loss/FRt1/detection_loss‚2.4371716961628054 epoch‚6 trainer/global_step‚8149 _runtime‚6353.212699644 _step‚669«†‘cœ Ý epoch‚6 trainer/global_step‚8149 _runtime‚6353.212699644 _step‚669 2loss/SRqqt1/assignment_loss‚1.4930928233404492 .loss/FRt3/detection_loss‚2.073513918647579 1loss/SRqqt1/detection_loss‚1.8910680096163306 /loss/FBt4/detection_loss‚2.7350150412983365 0loss/FRt3/assignment_loss‚3.1978373118475374 1loss/FBt4/assignment_loss‚0.14245404402449444 .loss/FBt3/detection_loss‚4.570625305175781 2loss/SRqqt2/assignment_loss‚0.8187664287656339 2loss/SRqqt3/assignment_loss‚1.3725946648961067 1loss/FBt3/assignment_loss‚0.21068098465889307 .loss/FRt4/detection_loss‚2.796214919537306 1loss/SRqqt2/detection_loss‚0.8928631377509899 .loss/FBt1/detection_loss‚2.857931411266327 /loss/FRt4/assignment_loss‚3.891396563500166 % _timestamp‚1.7884542616583607e+09 0loss/FRt1/assignment_loss‚2.9274124474060246 /loss/FRt2/detection_loss‚1.4839726212349804 1loss/SRqqt3/detection_loss‚1.6389493042869228 0loss/SRqqt4/detection_loss‚2.430712789297104 0loss/FRt2/assignment_loss‚0.9722513697364114 2loss/SRqqt4/assignment_loss‚1.7500951886177063 2loss/FBt1/assignment_loss‚0.016671459882581986 /loss/FBt2/detection_loss‚0.7486499840734506 &loss/total_loss‚43.535134342405485 0loss/FBt2/assignment_loss‚0.1851929173815751 /loss/FRt1/detection_loss‚2.4371716961628054^õ…š“ ‚(:ˆ üÊæÔÏœÓÕ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ ~Fž ‚(:» üÊæÔ„ê¿Ø" disk.nvme0c0n1.in‚ 190.51953125 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 246.6484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 3.21484375 disk.sda.out‚ 246.6484375) proc.memory.percent‚17.07141399383545 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚171.6474609375 disk.nvme0n1.in‚ 190.51953125 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.51953125% disk./.usageGB‚1109.5509147644043 disk.sda.in‚ 0.01171875$ memory_percent‚93.71871948242188) disk./.usagePercent‚74.46289286506592 disk.dm-1.out‚ 246.6484375 cpu‚25.348030866776977 network.recv‚ 2.276778e+06 disk.rbd0.in‚0# disk.dm-0.out‚27097.96923828125 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.9261988e+07( disk.nvme0c0n1.out‚27097.96923828125& disk.nvme0n1.out‚27097.96923828125$ proc.memory.availableMB‚1029.125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2796.98046875 proc.cpu.threads‚17@±Ã†žŸ ‚(:“ ‹ËæÔ¨¶ëÕ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚139.027' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.805399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?Âà?  ‚(:´ ‹ËæÔÙÍÅØ) disk./.usagePercent‚74.47974712108262 disk.rbd1.out‚ 3.21484375 disk.sda3.out‚247.375( disk.nvme0c0n1.out‚27098.26611328125 network.sent‚ 1.9369704e+07 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 190.51953125# disk.dm-0.out‚27098.26611328125) proc.memory.percent‚17.07141399383545 disk.sda.out‚247.375 disk.dm-1.in‚ 0.01171875$ memory_percent‚93.71569156646729 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 190.51953125& disk.nvme0n1.out‚27098.26611328125) proc.memory.availableMB‚ 1029.62109375 cpu‚25.347332073466344% disk./.usageGB‚1109.8020553588867 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 190.51953125 network.recv‚ 2.282439e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚247.375# proc.memory.rssMB‚ 2796.98046875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd2.out‚173.2216796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0ù“úq2¡ - _wandbruntime‚6388  _runtime‚6388Á©²È¢  $ _timestamp‚1.788454296718345e+09 _runtime‚6388.272742574 _step‚670 ' lr-AdamW/pg1‚5.2281363053553085e-05 ' lr-AdamW/pg2‚5.2281363053553085e-05 trainer/global_step‚8199ROÃYÍ£ Ç $ _timestamp‚1.788454296718345e+09 _runtime‚6388.272742574 _step‚670 ' lr-AdamW/pg1‚5.2281363053553085e-05 ' lr-AdamW/pg2‚5.2281363053553085e-05 trainer/global_step‚8199že Q®2¤ - _wandbruntime‚6388  _runtime‚6388NÖ‡«ç'¥ áO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476033581925888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108543073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39100793 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197422080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779347968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108542605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197422080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119023464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463570595066880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541059584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470405557115748 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154658758 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104982528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚8199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414LC¤äè¦ æO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108542605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197422080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119023464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463570595066880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541059584 XKDeviceStatsMonitor.on_train_b>‹ªatch_start/inactive_split.large_pool.allocated‚69441814 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470405557115748 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154658758 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104982528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚8199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476033581925888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108543073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39100793 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197422080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779347968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844Ÿ}Èä2§ - _wandbruntime‚6388  _runtime‚6388¡m©ð&¨ êM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154657174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7196626292224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788454297387268e+09 _step‚672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6861016863232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39105808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69449982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108555790 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470460606204156 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273694654 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132174158336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119037480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39105358 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482950951827968 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108555338 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414a ¾Äõ&© ïM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154677238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6861016863232 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6861016863232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39105808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69449982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108555790 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470460606204156 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273694654 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚527132174158336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119037480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39105358 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482950951827968 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108555338 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154657174 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7196626292224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788454297387268e+09 _step‚672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485751742730752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚6424.508205784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 % _timestamp‚1.7884543329535053e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119724264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473190927506040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69850214 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526865408 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155564772 _step‚675 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7238876624384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39329907 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109180119 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195120988160 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155564772 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239048636928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109180565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39330351‚ì'¸ æO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 % _timestamp‚1.7884543329535053e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119724264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473190927506040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69850214 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526865408 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155564772 _step‚675 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7238876624384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119724264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39329907 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478851215865344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109180119 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195120988160 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155564772 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239048636928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109180565 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39330351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473191598983156 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155584810 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478851233868288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485751742730752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚6424.508205784£Î Û2¹ - _wandbruntime‚6425  _runtime‚64256þ+«ð&º êM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155583226 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6425.189742782 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002669056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155583226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239906270208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109192821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6875942346124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485808915197440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530257641955328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478907568904192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109193264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788454333634633e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523017919041536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257825311744 trainer/global_step‚8249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39334882 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239746853888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39334441 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0²aÝÐ » ïM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119738282 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485809589576192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002669056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155583226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7239906270208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119738282 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_enHÃÙ%d/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109192821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6875942346124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466370705725440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485808915197440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530257641955328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478907568904192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109193264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788454333634633e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523017919041536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530257825311744 trainer/global_step‚8249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39334882 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7239746853888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39334441 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155583226 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6425.189742782 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6460.556047493 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380478976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39559934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156510862 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156490824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481668867807744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7281940168704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533320721415680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚679 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169033728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054844416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526038781246976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6941512671232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475977640850564 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276935928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320919465472 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7281940168704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533320721415680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚679 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6942169033728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156490824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488611054844416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526038781246976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6941512671232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚475977640850564 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276935928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533320919465472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/resKO3erved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6460.556047493 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488610380478976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39559934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156510862 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156490824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0§ÀgÅ™2É - _wandbruntime‚6461  _runtime‚6461‰×Ù¬ñ&Ê ëM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948358 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70266782 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383424258560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6942332099072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533383236707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7282968333824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488668227324416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39564991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109831321 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 trainer/global_step‚8299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39564541 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6942332099072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109831773 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884543696781776e+09 _step‚680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚6461.233045368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120439082 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 Eö&Ë ðM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120439082 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70266782 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533383424258560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6942332099072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70266780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481725220846592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533383236707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7282968333824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488668227324416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39564991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109831321 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 trainer/global_step‚8299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39564541 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6942332099072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109831773 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884543696781776e+09 _step‚680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚6461.233045368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481725238849536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¨\þnF2Ì - _wandbruntime‚6461  _runtime‚6461ÿîS[Í Õ /loss/FRt4/detection_loss‚3.0051246904409847 0loss/FRt1/assignment_loss‚2.7972619940594927 0loss/FBt4/assignment_loss‚0.7388728625540223 .loss/FBt1/detection_loss‚2.102886804512569 /loss/FBt3/detection_loss‚2.1500717103481293 /loss/FBt4/detection_loss‚3.4843623638153076 epoch‚6 trainer/global_step‚8299 _runtime‚6461.234276898 /loss/FRt2/detection_loss‚1.8380813531234468 /loss/FRt4/assignment_loss‚3.931055261538579 2loss/SRqqt3/assignment_loss‚1.6178363954510402 0loss/FBt3/assignment_loss‚0.5149753052508459 % _timestamp‚1.7884543696799836e+09 _step‚681 .loss/FRt1/detection_loss‚2.490748156861561 1loss/SRqqt1/detection_loss‚1.5580485116262905 /loss/FBt2/detection_loss‚0.6704596283907566 /loss/FRt3/assignment_loss‚2.772776082805965 1loss/SRqqt1/assignment_loss‚1.370234850313148 2loss/SRqqt4/assignment_loss‚2.0933768238985175 1loss/FBt1/assignment_loss‚0.21712844532781414 /loss/FRt3/detection_loss‚2.2631553068109183 %loss/total_loss‚43.44886931851735 2loss/SRqqt2/assignment_loss‚0.9204235725889081 1loss/FBt2/assignment_loss‚0.14045666308235014 0loss/FRt2/assignment_loss‚1.6029197636403536 0loss/SRqqt4/detection_loss‚2.346586621168888 1loss/SRqqt2/detection_loss‚1.0676551116140265 1loss/SRqqt3/detection_loss‚1.7543710392934304}{4E`Î Ú 1loss/FBt2/assignment_loss‚0.14045666308235014 0loss/FRt2/assignment_loss‚1.6029197636403536 0loss/SRqqt4/detection_loss‚2.346586621168888 1loss/SRqqt2/detection_loss‚1.0676551116140265 1loss/SRqqt3/detection_loss‚1.7543710392934304 /loss/FRt4/detection_loss‚3.0051246904409847 0loss/FRt1/assignment_loss‚2.7972619940594927 0loss/FBt4/assignment_loss‚0.7388728625540223 .loss/FBt1/detection_loss‚2.102886804512569 /loss/FBt3/detection_loss‚2.1500717103481293 /loss/FBt4/detection_loss‚3.4843623638153076 epoch‚6 trainer/global_step‚8299 _runtime‚6461.234276898 /loss/FRt2/detection_loss‚1.8380813531234468 /loss/FRt4/assignment_loss‚3.931055261538579 2loss/SRqqt3/assignment_loss‚1.6178363954510402 0loss/FBt3/assignment_loss‚0.5149753052508459 % _timestamp‚1.7884543696799836e+09 _step‚681 .loss/FRt1/detection_loss‚2.490748156861561 1loss/SRqqt1/detection_loss‚1.5580485116262905 /loss/FBt2/detection_loss‚0.6704596283907566 /loss/FRt3/assignment_loss‚2.772776082805965 1loss/SRqqt1/assignment_loss‚1.370234850313148 2loss/SRqqt4/assignment_loss‚2.0933768238985175 1loss/FBt1/assignment_loss‚0.21712844532781414 /loss/FRt3/detection_loss‚2.2631553068109183 %loss/total_loss‚43.44886931851735 2loss/SRqqt2/assignment_loss‚0.9204235725889081©榣’Ï ‚(:‡ åËæÔ¢®ÄÕ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡úHOÐ ‚(:Ä åËæÔ¨ò–Ø disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2796.98046875 cpu‚25.343250774889693 disk.nvme0n1.in‚ 190.51953125 disk.drbd1777.out‚0 disk.dm-0.in‚ 190.51953125 disk.dm-1.out‚ 251.25390625 disk.rbd1.out‚ 3.373046875 network.recv‚ 2.311333e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 251.25390625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1029.39453125 disk.rbd0.out‚0# disk.dm-0.out‚27495.47314453125" disk.nvme0c0n1.in‚ 190.51953125 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.51360590953865% disk./.usageGB‚1110.3065757751465 disk.rbd2.in‚0 disk.rbd2.out‚178.3154296875$ memory_percent‚93.71707439422607) proc.memory.percent‚17.07141399383545 network.sent‚ 1.9610187e+07& disk.nvme0n1.out‚27495.47314453125 disk.drbd1777.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27495.47314453125 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda.out‚ 251.25390625A«4“Ñ ‚(:ˆ ôËæÔ”ëºÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ïl©JÒ ‚(:¿ ôËæÔªãšØ cpu‚25.34253591097444 disk.sda3.out‚ 251.9296875) disk./.usagePercent‚74.51356034019756 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 251.9296875 disk.sda1.in‚0) proc.memory.percent‚17.07141399383545 disk.drbd1777.in‚0 disk.dm-0.in‚ 190.51953125 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.9719011e+07 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 190.51953125 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0 network.recv‚ 2.317927e+06% disk./.usageGB‚1110.3058967590332 disk.dm-1.out‚ 251.9296875 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚27496.76708984375 disk.nvme0n1.in‚ 190.51953125& disk.nvme0n1.out‚27496.76708984375 disk.drbd1777.out‚0 disk.rbd2.out‚179.9912109375# memory_percent‚93.7185525894165 disk.rbd1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2796.98046875# disk.dm-0.out‚27496.76708984375) proc.memory.availableMB‚ 1029.152343756þ¹žÓ ‚(:“ ƒÌæÔÛâÉÕ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.636599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X€î1KÔ ‚(:À ƒÌæÔñçØ disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2796.98046875) proc.memory.percent‚17.07141399383545 proc.cpu.threads‚17 network.sent‚ 1.9724378e+07 network.recv‚ 2.321706e+06 disk.sda3.out‚ 252.9765625 disk.rbd2.in‚0 disk.rbd2.out‚179.9912109375 disk.dm-0.in‚ 193.08984375# disk.dm-0.out‚27596.66357421875 disk.rbd1.out‚ 3.373046875 disk.rbd3.in‚0 disk.drbd1777.in‚0 cpu‚25.341858197939295 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875% disk./.usageGB‚1110.3054695129395 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚74.51353166735373" disk.nvme0c0n1.in‚ 193.08984375& disk.nvme0n1.out‚27596.66357421875 disk.drbd1777.out‚0$ memory_percent‚93.71857643127441 disk.rbd3.out‚0 disk.sda.out‚ 252.9765625 disk.nvme0n1.in‚ 193.08984375 disk.dm-1.out‚ 252.9765625( disk.nvme0c0n1.out‚27596.66357421875 disk.sda2.out‚0( proc.memory.availableMB‚ 1029.1484375ýÀÃD2Õ - _wandbruntime‚6496  _runtime‚6496'íÝÇÖ Á & lr-AdamW/pg1‚4.872205809673003e-05 & lr-AdamW/pg2‚4.872205809673003e-05 trainer/global_step‚8349 % _timestamp‚1.7884544053765504e+09 _runtime‚6496.930956488 _step‚682òvÜÌ× Æ _runtime‚6496.930956488 _step‚682 & lr-AdamW/pg1‚4.872205809673003e-05 & lr-AdamW/pg2‚4.872205809673003e-05 trainer/global_step‚8349 % _timestamp‚1.7884544053765504e+09ª4:2Ø - _wandbruntime‚6496  _runtime‚6496zä‡Ö Ù àO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325001278976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8349 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498477056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436914 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478763011240856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6496.9— À31830645 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚529121318130176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446513264640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416876 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6956796589976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154839552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536446319409152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39789217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325171194368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788454405377293e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491469018227200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39788755 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436914 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478763682717972 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6957451027732 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484486519750144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536446513264640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416876 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484486537753088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6956796589976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157416876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154839552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536446319409152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39789217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325171194368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788454405377293e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491469018227200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39788755 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157436914 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478763682717972 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6957451027732 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325001278976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8349 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0«ÛHvb2Û - _wandbruntime‚6497  _runtime‚6497¶Š«Ìñ&Ü ëM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484542890791936 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110468490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39793310 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536508818536448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471861135820800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484542872788992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278575210 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚684 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455394 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478818731806380 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317904896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚8349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455394 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509012378624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157435330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110468949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325849668608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 % _timestamp‚1.7884544060619998e+09“¼Hq¿Ý ðM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚529182968867840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157455394 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478818731806380 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6983317904896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚8349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455394 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509012378624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157435330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110468949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7325849668608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚529182992807936 % _timestamp‚1.7884544060619998e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484542890791936 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110468490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 F@DeviceStatsMonitor.on_train_b¾„Ýt7atch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39793310 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536508818536448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471861135820800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484542872788992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278575210 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚684 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F2è - _wandbruntime‚6533  _runtime‚6533Ù! æ'é àO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121826664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40016691 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40017128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330340864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚8399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367140 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487304189695488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111092103 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6533.658198458 _step‚687 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494328330340864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111092542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571965629440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585380 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532203855013376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788454442103609e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7368270045696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487304189695488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111092103 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6533.658198458 _step‚687 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71075414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494328330340864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111092542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539571965629440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481549724585380 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532203855013376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788454442103609e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7368270045696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40017128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494328330340864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚8399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6997631929384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367140¯ØT 2ë - _wandbruntime‚6534  _runtime‚6534¹á~Àì êM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369143035904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368992008192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364348 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71083580 cODeviceStatsMonitor.on_train_batch_end/inactive_sï­~.0!plit_bytes.large_pool.allocated‚532265529691136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361382 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960086528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303710720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111104822 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40021242 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634497759232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6534.336935133 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960086528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158381446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚8399 jüõ&í ïM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 trainer/global_step‚8399 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7369143035904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7368992008192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364348 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71083580 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532265529691136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158361382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361382 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7024960086528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303710720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111104822 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40021242 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539634497759232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6534.336935133 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960086528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158381446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0°‡AÿÔ2î - _wandbruntime‚6534  _runtime‚6534«*€„Wï Ñ 0loss/FRt4/assignment_loss‚3.7739691957831383 1loss/SRqqt1/assignment_loss‚1.032252785570633 .loss/FRt4/detection_loss‚2.717009287327528 1loss/SRqqt1/detection_loss‚1.7455733730815923 /loss/FRt3/assignment_loss‚3.161834837617101 .loss/FRt3/detection_loss‚2.360283039711617 _runtime‚6534.338678107 0loss/FRt2/assignment_loss‚1.0951254963874817 1loss/SRqqt2/assignment_loss‚0.943270718678832 1loss/SRqqt3/detection_loss‚1.5499372972683474 2loss/SRqqt4/assignment_loss‚2.1667286343872547 0loss/FBt2/assignment_loss‚0.3505743192441416 1loss/FBt3/assignment_loss‚0.45551763016443986 /loss/FRt2/detection_loss‚1.9638570299372078 _step‚689 0loss/SRqqt2/detection_loss‚1.036622035666369 /loss/FRt1/assignment_loss‚2.412916591633921 /loss/FRt1/detection_loss‚2.2514057036327277 .loss/FBt1/detection_loss‚3.007829968745892 epoch‚6 trainer/global_step‚8399 1loss/SRqqt3/assignment_loss‚1.546132618188858 /loss/FBt2/detection_loss‚0.4985441080674398 .loss/FBt3/detection_loss‚2.496651997933021 %loss/total_loss‚41.67074678911516 1loss/FBt1/assignment_loss‚0.08115844516970146 1loss/FBt4/assignment_loss‚0.09081756789237207 1loss/SRqqt4/detection_loss‚2.2287036161869764 /loss/FBt4/detection_loss‚2.7040304908385644 $ _timestamp‚1.788454442784335e+09‰ þ¬\ð Ö 1loss/FBt1/assignment_loss‚0.08115844516970146 1loss/FBt4/assignment_loss‚0.09081756789237207 1loss/SRqqt4/detection_loss‚2.2287036161869764 /loss/FBt4/detection_loss‚2.7040304908385644 $ _timestamp‚1.788454442784335e+09 0loss/FRt4/assignment_loss‚3.7739691957831383 1loss/SRqqt1/assignment_loss‚1.032252785570633 .loss/FRt4/detection_loss‚2.717009287327528 1loss/SRqqt1/detection_loss‚1.7455733730815923 /loss/FRt3/assignment_loss‚3.161834837617101 .loss/FRt3/detection_loss‚2.360283039711617 _runtime‚6534.338678107 0loss/FRt2/assignment_loss‚1.0951254963874817 1loss/SRqqt2/assignment_loss‚0.943270718678832 1loss/SRqqt3/detection_loss‚1.5499372972683474 2loss/SRqqt4/assignment_loss‚2.1667286343872547 0loss/FBt2/assignment_loss‚0.3505743192441416 1loss/FBt3/assignment_loss‚0.45551763016443986 /loss/FRt2/detection_loss‚1.9638570299372078 _step‚689 0loss/SRqqt2/detection_loss‚1.036622035666369 /loss/FRt1/assignment_loss‚2.412916591633921 /loss/FRt1/detection_loss‚2.2514057036327277 .loss/FBt1/detection_loss‚3.007829968745892 epoch‚6 trainer/global_step‚8399 1loss/SRqqt3/assignment_loss‚1.546132618188858 /loss/FBt2/detection_loss‚0.4985441080674398 .loss/FBt3/detection_loss‚2.496651997933021 %loss/total_loss‚41.67074678911516±V’Ïñ ‚(:’ °ÌæÔ™öê× gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.287999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏGMmIò ‚(:¾ °ÌæÔñÿ¡Ù proc.cpu.threads‚17 disk.sda1.out‚0) disk./.usagePercent‚74.51360514151605 disk.rbd2.out‚183.4716796875% disk.nvme0n1.out‚27795.7099609375 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0 disk.sda3.out‚ 254.94140625$ memory_percent‚93.71917247772217 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2796.98046875) proc.memory.availableMB‚ 1029.05078125) proc.memory.percent‚17.07141399383545 cpu‚25.339854134373784 disk.drbd1777.in‚0" disk.dm-0.out‚27795.7099609375 disk.sda.out‚ 254.94140625 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 194.2265625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 network.sent‚ 1.9844983e+07 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.2265625' disk.nvme0c0n1.out‚27795.7099609375 disk.nvme0n1.in‚ 194.2265625 disk.sda1.in‚0 network.recv‚ 2.336095e+06% disk./.usageGB‚1110.3065643310547 disk.rbd0.in‚0 disk.dm-1.out‚ 254.94140625 disk.rbd3.in‚0 ó ‚(:ˆ ¿ÌæÔÔîÏÕ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Ë<Jô ‚(:¿ ¿ÌæÔàŒÈØ disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 255.58984375 disk.dm-0.in‚ 194.9453125 disk.sda.in‚ 0.01171875 cpu‚25.339160990056328 network.sent‚ 1.9952711e+07 network.recv‚ 2.341705e+06& disk.nvme0n1.out‚27796.24853515625 disk.dm-1.out‚ 255.58984375 disk.sda3.in‚ 0.01171875) proc.memory.percent‚17.07141399383545# proc.memory.rssMB‚ 2796.98046875 disk.sda1.in‚0) disk./.usagePercent‚74.51362075797564$ disk./.usageGB‚1110.306797027588 disk.rbd0.in‚0 disk.rbd2.out‚185.0458984375! disk.nvme0c0n1.in‚ 194.9453125# disk.dm-0.out‚27796.24853515625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0$ memory_percent‚93.71762275695801( proc.memory.availableMB‚ 1029.3046875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚27796.24853515625 disk.nvme0n1.in‚ 194.9453125 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 255.58984375àŽŠŸ2õ - _wandbruntime‚6569  _runtime‚6569°;ºÒÆö À % _timestamp‚1.7884544779280295e+09 _runtime‚ 6569.48245282 _step‚690 & lr-AdamW/pg1‚4.639401940821935e-05 & lr-AdamW/pg2‚4.639401940821935e-05 trainer/global_step‚8449ëîÂË÷ Å _runtime‚ 6569.48245282 _step‚690 & lr-AdamW/pg1‚4.639401940821935e-05 & lr-AdamW/pg2‚4.639401940821935e-05 trainer/global_step‚8449 % _timestamp‚1.7884544779280295e+09²bzÆá2ø - _wandbruntime‚6569  _runtime‚6569M^žBÜù áO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281816484 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497186968089088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7038467268792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542697781858304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159289018 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490121823634944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411204494848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7065126451200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293723648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚6569.483501466 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535286391896576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7064470088704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111730407 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796444 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7064470088704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40246593 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40246134 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697596391424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smal^ ¸ã l_pool.allocated‚ 7411366021632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7065126451200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484335094975672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚8449 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111729946 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159268980 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490121841637888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159289018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430BƒQœì'ú æO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542697781858304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159289018 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490121823634944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7411204494848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7065126451200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293723648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚6569.483501466 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535286391896576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7064470088704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111730407 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796444 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7064470088704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40246593 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40246134 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542697596391424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7411366021632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7065126451200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122527464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484335094975672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚8449 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111729946 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159268980 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490121841637888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159289018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281816484 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497186968089088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7038467268792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040³Ù_˜žû ‚(:“ ÎÌæÔÆŸØÕ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.802999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S»?1Cü ‚(:¸ ÎÌæÔô–©Ø disk.rbd1.in‚0 disk.sda3.out‚ 256.08984375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 194.9453125$ disk.nvme0n1.out‚27893.966796875 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.9453125& proc.memory.availableMB‚ 1029.09375 network.sent‚ 1.9958965e+07) disk./.usagePercent‚74.51362229402085 disk.sda1.out‚0& disk.nvme0c0n1.out‚27893.966796875 disk.dm-1.out‚ 256.08984375) proc.memory.percent‚17.07141399383545 proc.cpu.threads‚17 network.recv‚ 2.346393e+06 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 194.9453125 disk.drbd1777.in‚0$ memory_percent‚93.71891021728516! disk.dm-0.out‚27893.966796875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 256.08984375 disk.dm-1.in‚ 0.01171875 cpu‚25.338543228732373% disk./.usageGB‚1110.3068199157715 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚185.1474609375 disk.sda2.out‚0# proc.memory.rssMB‚ 2796.98046875qÌ92ý - _wandbruntime‚6570  _runtime‚6570cÄ®˜î&þ èM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490178194676736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71491982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7039938141720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484390815541196 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703756 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71491980 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945892352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111743048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8449 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚692 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7412231464960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 6570.1762071 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40250617 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497243466190336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111742597 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414Iu×!ÿ íM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71491980 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945892352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111743048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8449 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚692 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7412231464960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚ 6570.1762071 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40250617 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497243466190336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111742597 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490178194676736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71491982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7039938141720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.all n’ñocated‚477351548876800 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484390815541196 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7039283703756 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0´Þì2€!- _wandbruntime‚6570  _runtime‚6570|½WàW!Ñ .loss/FRt1/detection_loss‚2.111797571816343 epoch‚6 % _timestamp‚1.7884544786229844e+09 .loss/FBt4/detection_loss‚2.766316533088684 _runtime‚ 6570.17724214 1loss/FBt2/assignment_loss‚0.11593596020478829 .loss/FRt2/detection_loss‚1.470803049039738 -loss/FRt3/detection_loss‚1.86766203906801 .loss/FRt4/detection_loss‚2.957241489337041 1loss/SRqqt3/detection_loss‚1.7472003120183945 /loss/FRt1/assignment_loss‚2.608409991923799 1loss/FBt1/assignment_loss‚0.02293631146673241 /loss/FBt3/detection_loss‚2.8453039824962616 trainer/global_step‚8449 _step‚693 0loss/FRt4/assignment_loss‚3.9640726768053494 2loss/SRqqt4/assignment_loss‚2.1838042996823788 0loss/FBt3/assignment_loss‚0.5681187365844371 .loss/FBt1/detection_loss‚2.681593179702759 /loss/FBt2/detection_loss‚0.6909182384023974 %loss/total_loss‚42.11041619479302 0loss/FRt2/assignment_loss‚1.4482425471832012 2loss/SRqqt1/assignment_loss‚1.2684559626504779 1loss/SRqqt2/assignment_loss‚1.002065137683137 2loss/SRqqt3/assignment_loss‚1.2468200527504087 1loss/SRqqt1/detection_loss‚1.5418093836668767 1loss/SRqqt4/detection_loss‚2.4801413491368294 /loss/FRt3/assignment_loss‚3.404785595558308 1loss/FBt4/assignment_loss‚0.16017150358917812 1loss/SRqqt2/detection_loss‚0.9558102909374881Æ;9\‚!Ö .loss/FBt4/detection_loss‚2.766316533088684 _runtime‚ 6570.17724214 1loss/FBt2/assignment_loss‚0.11593596020478829 .loss/FRt2/detection_loss‚1.470803049039738 -loss/FRt3/detection_loss‚1.86766203906801 .loss/FRt4/detection_loss‚2.957241489337041 1loss/SRqqt3/detection_loss‚1.7472003120183945 /loss/FRt1/assignment_loss‚2.608409991923799 1loss/FBt1/assignment_loss‚0.02293631146673241 /loss/FBt3/detection_loss‚2.8453039824962616 trainer/global_step‚8449 _step‚693 0loss/FRt4/assignment_loss‚3.9640726768053494 2loss/SRqqt4/assignment_loss‚2.1838042996823788 0loss/FBt3/assignment_loss‚0.5681187365844371 .loss/FBt1/detection_loss‚2.681593179702759 /loss/FBt2/detection_loss‚0.6909182384023974 %loss/total_loss‚42.11041619479302 0loss/FRt2/assignment_loss‚1.4482425471832012 2loss/SRqqt1/assignment_loss‚1.2684559626504779 1loss/SRqqt2/assignment_loss‚1.002065137683137 2loss/SRqqt3/assignment_loss‚1.2468200527504087 1loss/SRqqt1/detection_loss‚1.5418093836668767 1loss/SRqqt4/detection_loss‚2.4801413491368294 /loss/FRt3/assignment_loss‚3.404785595558308 1loss/FBt4/assignment_loss‚0.16017150358917812 1loss/SRqqt2/detection_loss‚0.9558102909374881 .loss/FRt1/detection_loss‚2.111797571816343 epoch‚6 % _timestamp‚1.7884544786229844e+09µîw¾žƒ!‚(:“ ÝÌæÔܘûÚ gpu.0.gpu‚57.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚152.266' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.453199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=X¤ÝD„!‚(:¹ ÝÌæÔ¯é°Þ disk.sda3.out‚ 256.7265625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 2.006852e+07 disk.rbd0.in‚0' proc.memory.availableMB‚ 950.4765625) disk./.usagePercent‚74.51362434208112( disk.nvme0c0n1.out‚27991.59033203125 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.rbd1.out‚ 3.373046875 cpu‚25.34230235524289 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 194.9453125 disk.nvme0n1.in‚ 194.9453125& disk.nvme0n1.out‚27991.59033203125$ memory_percent‚94.19875144958496# proc.memory.rssMB‚ 2875.26953125( proc.memory.percent‚17.5492525100708 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.out‚186.7216796875 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 256.7265625 network.recv‚ 2.352621e+06 disk.rbd3.in‚0 disk.sda.out‚ 256.7265625 disk.sda1.out‚0 disk.dm-0.in‚ 194.9453125# disk.dm-0.out‚27991.59033203125% disk./.usageGB‚1110.3068504333496ÍÈ«’’…!‚(:‡ ìÌæÔ¡ÑéÕ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìw:†!‚(:¯ ìÌæÔ¢›•Ø! disk.dm-0.out‚27992.333984375 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0 network.sent‚ 2.0153843e+07% disk./.usageGB‚1110.3069152832031 disk.rbd0.out‚0 disk.nvme0n1.in‚ 194.9453125 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.sda.out‚ 257.5078125 cpu‚25.35338737733984 proc.cpu.threads‚17 network.recv‚ 2.35905e+06( disk./.usagePercent‚74.5136286942092 disk.sda3.out‚ 257.5078125 disk.dm-0.in‚ 194.9453125# memory_percent‚94.1995620727539% proc.memory.availableMB‚ 950.34375 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2875.26953125( proc.memory.percent‚17.5492525100708 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚186.8857421875! disk.nvme0c0n1.in‚ 194.9453125$ disk.nvme0n1.out‚27992.333984375 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 257.5078125 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚27992.333984375 disk.sda2.out‚0ÜÕ¨“‡!‚(:ˆ ûÌæÔ‡—ÙÕ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.564' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤~Aˆ!‚(:¶ ûÌæÔœ¼ƒØ disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0$ disk.nvme0n1.out‚28089.830078125 disk.sda2.out‚0 disk.sda.out‚ 258.19140625 network.recv‚ 2.362803e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 194.9453125 disk.dm-0.in‚ 194.9453125 disk.dm-1.out‚ 258.19140625 disk.rbd0.in‚0 disk.rbd2.out‚188.4599609375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 258.19140625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd3.in‚0$ memory_percent‚94.20022964477539 cpu‚25.363884330710615 network.sent‚ 2.0159174e+07) disk./.usagePercent‚74.51363253432221! disk.nvme0c0n1.in‚ 194.9453125& disk.nvme0c0n1.out‚28089.830078125& proc.memory.availableMB‚ 950.234375( proc.memory.percent‚17.5492525100708$ disk./.usageGB‚1110.306972503662 disk.sda1.out‚0! disk.dm-0.out‚28089.830078125# proc.memory.rssMB‚ 2875.26953125 disk.dm-1.in‚ 0.01171875š2‰!- _wandbruntime‚6627  _runtime‚6627`è‹M*Š!ÇT E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚6627.605023766 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 123253184 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7441512334336 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788454536050263e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40531215 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8462 _step‚694 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112448077 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7441396990464 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚486862574589936 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157891 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7094012577792 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40531059 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446949440000 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚479794696788992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904707 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112447919 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 123253182 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788454536050263e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40531215 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8462 _step‚694 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112448077 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7441396990464 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚486862574589936 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 123253184 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157891 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7094012577792 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚71916862 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40531059 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545446949440000 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚479794696788992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904707 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112447919 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 123263988 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377704448 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚40535825 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚538043288899072 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚695 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚156 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚499847156156416 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7094377704448 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7441762117120 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545485051016192 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912009 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160020 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚499846645323264 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7094870534656 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870534656 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 123263988 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377704448 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚40535825 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚538043288899072 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚695 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚156 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚499847156156416 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7094377704448 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7441762117120 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545485051016192 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912009 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160020 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚499846645323264 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7094870534656 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870534656 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895002841949463 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.579325258731842 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26832398772239685 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.533917248249054 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5922339558601379 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5261493921279907 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5022744536399841 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.11636904627084732 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5645439028739929 5 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.27182537317276 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5542751550674438 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012186380103230476 )loss/val_total_loss‚42.93116213463139 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.044243622571229935 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.359523206949234 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5412354469299316 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18390578031539917 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_5‚0.10890937596559525 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44428130984306335 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14750082790851593 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2635858654975891 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5854120850563049 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14726917445659637 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14666666090488434 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14604395627975464 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5706075429916382 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09009556472301483 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909354448318481 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1451665312051773 *jet/accuracy_2_of_6‚0.3589743971824646 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4397279918193817 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5614582896232605 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5478426218032837 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_4_of_4‚0.0023547881282866 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5887928605079651 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017101285979151726 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4950871467590332 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.46026936173439026 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5506916642189026 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31366169452667236 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991314709186554 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07270482182502747 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5109658241271973 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24404795467853546 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5681172609329224 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29704034328460693 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1944444477558136 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 % _timestamp‚1.7884545494430048e+09 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6005747318267822 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.47385624051094055 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9814580678939819 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5417710542678833 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885948896408081 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4943249821662903 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.624404788017273 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48037636280059814 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5061473250389099 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4600600302219391 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 *jet/accuracy_1_of_4‚0.9188240170478821 ,particle/sensitivity‚0.34764060378074646 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111342906952 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03888293728232384 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.976670503616333 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 *jet/accuracy_3_of_6‚0.1538461595773697 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09188899397850037 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 _step‚696 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27283915877342224 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39441773295402527 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12894736230373383 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 jet/accuracy_6_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5449687242507935 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12687771022319794 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10596708208322525 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚ʹ´$c0 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.45120424032211304 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851258397102356 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004534178413450718 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.25674161314964294 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8183322548866272 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977625012397766 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16746141016483307 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15789835155010223 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17326101660728455 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02164509892463684 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0018518519354984164 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3361110985279083 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.042101286351680756 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9871526956558228 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6134872436523438 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13543082773685455 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4147877097129822 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17353565990924835 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6983782649040222 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9836439490318298 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2907407581806183 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9672362804412842 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5327224731445312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14926667511463165 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9876959919929504 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5654882788658142 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.534350574016571 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 epoch‚6 *jet/accuracy_1_of_2‚0.7765482068061829 particle/accuracy_1_of_2‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5594992637634277 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19212837517261505 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013492063619196415 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4085565507411957 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1542285680770874 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19586825370788574 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1959804743528366 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5652542114257812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5603620409965515 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2858239710330963 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4740740954875946 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5180780291557312 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5412354469299316 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9492185115814209 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1490756869316101 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 particle/accuracy_2_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 *jet/accuracy_2_of_5‚0.5027201771736145 +jet/accuracy_4_of_6‚0.07692307978868484 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14959536492824554 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21558204293251038 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.558782160282135 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5560468435287476 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 particle/accuracy_3_of_7‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23435048758983612 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5250580310821533 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20392771065235138 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11599213629961014 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5798066854476929 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26154837012290955 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2821572721004486 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9819572567939758 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 (particle/f_score‚0.42851725220680237 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5753968358039856 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9821417927742004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14702048897743225 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011652464978396893 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20802278816699982 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8640351295471191 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.522325336933136 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10554566979408264 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47174331545829773 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10357668250799179 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47476935386657715 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5148305296897888 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01774848997592926 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 (particle/accuracy‚0.8552878499031067 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2199178785085678 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6049382090568542 particle/accuracy_5_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16329385340213776 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.38419705629348755 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10164009034633636 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09856700897216797 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0022564581595361233 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33541664481163025 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9873627424240112 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5798583626747131 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1445402204990387 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5117320418357849 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15311112999916077 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015326529741287 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8845165967941284 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995271921157837 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.403917133808136 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5683127641677856 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5332256555557251 jet/accuracy_7_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 trainer/global_step‚8462 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3910371959209442 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4376780688762665 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3785136938095093 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_3_of_3‚0.02635762467980385 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02380952425301075 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _runtime‚6640.999878409 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5707985162734985 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6424950957298279 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5701273083686829 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2599140703678131 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3565565049648285 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9723419547080994 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5470210313796997 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17082586884498596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5180225372314453 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29260194301605225 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21846845746040344 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19909898936748505 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03150954470038414 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38531747460365295 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11963718384504318 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17326101660728455 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20037752389907837 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38923612236976624 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20833337306976318 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13837772607803345 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23275862634181976 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04885057732462883 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5739011168479919 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.590717613697052 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898621439933777 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3482687175273895 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 *jet/accuracy_1_of_1‚0.5042867064476013 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909922480583191 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037461597472429276 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4653022289276123 ("Purity/*FRt*SRqqt4FBt/event_purity‚0©cWΓ!ʦ % _timestamp‚1.7884545494430048e+09 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6005747318267822 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.47385624051094055 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9814580678939819 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5417710542678833 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885948896408081 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4943249821662903 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.624404788017273 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48037636280059814 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5061473250389099 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4600600302219391 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 *jet/accuracy_1_of_4‚0.9188240170478821 ,particle/sensitivity‚0.34764060378074646 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111342906952 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03888293728232384 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.976670503616333 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 *jet/accuracy_3_of_6‚0.1538461595773697 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09188899397850037 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 _step‚696 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27283915877342224 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39441773295402527 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12894736230373383 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 jet/accuracy_6_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5449687242507935 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12687771022319794 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10596708208322525 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.45120424032211304 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851258397102356 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004534178413450718 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.25674161314964294 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8183322548866272 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977625012397766 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16746141016483307 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15789835155010223 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17326101660728455 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02164509892463684 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0018518519354984164 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3361110985279083 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.042101286351680756 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9871526956558228 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6134872436523438 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13543082773685455 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4147877097129822 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17353565990924835 9"Purity/*FRt0SRqqt1FBt/event_puri“«o܃vty‚0.6983782649040222 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9836439490318298 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2907407581806183 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9672362804412842 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5327224731445312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14926667511463165 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9876959919929504 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5654882788658142 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.534350574016571 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 epoch‚6 *jet/accuracy_1_of_2‚0.7765482068061829 particle/accuracy_1_of_2‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5594992637634277 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19212837517261505 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013492063619196415 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4085565507411957 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1542285680770874 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19586825370788574 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1959804743528366 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5652542114257812 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5603620409965515 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2858239710330963 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4740740954875946 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5180780291557312 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5412354469299316 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9492185115814209 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1490756869316101 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 particle/accuracy_2_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 *jet/accuracy_2_of_5‚0.5027201771736145 +jet/accuracy_4_of_6‚0.07692307978868484 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14959536492824554 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21558204293251038 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.558782160282135 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5560468435287476 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 particle/accuracy_3_of_7‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23435048758983612 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5250580310821533 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20392771065235138 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11599213629961014 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5798066854476929 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26154837012290955 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2821572721004486 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9819572567939758 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 (particle/f_score‚0.42851725220680237 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5753968358039856 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9821417927742004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14702048897743225 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011652464978396893 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20802278816699982 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8640351295471191 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.522325336933136 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10554566979408264 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47174331545829773 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10357668250799179 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47476935386657715 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5148305296897888 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01774848997592926 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 (particle/accuracy‚0.8552878499031067 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2199178785085678 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6049382090568542 particle/accuracy_5_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16329385340213776 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.38419705629348755 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10164009034633636 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09856700897216797 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0022564581595361233 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33541664481163025 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9873627424240112 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5798583626747131 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1445402204990387 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5117320418357849 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15311112999916077 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11015326529741287 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8845165967941284 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995271921157837 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.403917133808136 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5683127641677856 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5332256555557251 jet/accuracy_7_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 trainer/global_step‚8462 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3910371959209442 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4376780688762665 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3785136938095093 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_3_of_3‚0.02635762467980385 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02380952425301075 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 _runtime‚6640.999878409 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5707985162734985 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6424950957298279 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5701273083686829 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2599140703678131 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3565565049648285 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9723419547080994 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5470210313796997 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17082586884498596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5180225372314453 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29260194301605225 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21846845746040344 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19909898936748505 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03150954470038414 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.38531747460365295 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11963718384504318 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17326101660728455 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20037752389907837 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38923612236976624 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20833337306976318 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13837772607803345 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23275862634181976 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04885057732462883 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5739011168479919 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.590717613697052 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898621439933777 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3482687175273895 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 *jet/accuracy_1_of_1‚0.5042867064476013 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909922480583191 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037461597472429276 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4653022289276123 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5697759389877319 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08121723681688309 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1269092708826065 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39534884691238403 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895002841949463 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.579325258731842 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26832398772239685 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.533917248249054 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5922339558601379 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5261493921279907 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5022744536399841 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.11636904627084732 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5645439028739929 5 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.27182537317276 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5542751550674438 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012186380103230476 )loss/val_total_loss‚42.93116213463139 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.044243622571229935 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.359523206949234 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5412354469299316 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18390578031539917 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_5‚0.10890937596559525 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44428130984306335 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14750082790851593 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2635858654975891 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5854120850563049 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14726917445659637 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14666666090488434 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14604395627975464 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5706075429916382 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09009556472301483 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909354448318481 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1451665312051773 *jet/accuracy_2_of_6‚0.3589743971824646 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4397279918193817 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5614582896232605 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5478426218032837 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_4_of_4‚0.0023547881282866 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5887928605079651 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017101285979151726 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4950871467590332 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.46026936173439026 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5506916642189026 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31366169452667236 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991314709186554 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07270482182502747 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5109658241271973 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24404795467853546 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5681172609329224 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29704034328460693 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1944444477558136 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875¸”ÌPÊ’”!‚(:‡ ™ÍæÔì¹ÙÕ gpu.0.gpu‚21.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚30.0 gpu.0.powerWatts‚76.284' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.2568 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tJ6nA•!‚(:¶ ™ÍæÔÎ¥·Ù! disk.nvme0c0n1.in‚ 194.9453125% disk.nvme0n1.out‚28188.5107421875 disk.rbd3.out‚0$ memory_percent‚95.14904022216797* proc.memory.percent‚20.440196990966797 cpu‚25.382849391575718 network.recv‚ 2.374126e+06) disk./.usagePercent‚74.54485495719062' disk.nvme0c0n1.out‚28188.5107421875 disk.sda2.out‚0% proc.memory.availableMB‚ 794.78125 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 194.9453125 disk.drbd1777.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1110.7722091674805 disk.sda1.out‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 3348.921875 disk.sda1.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚28188.5107421875 disk.dm-1.out‚ 259.5703125 disk.sda.out‚ 259.5703125 disk.rbd2.in‚0 disk.dm-0.in‚ 194.9453125 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 2.0273346e+07 disk.sda3.out‚ 259.5703125 disk.rbd2.out‚190.2138671875Ò棉“–!‚(:ˆ ¨ÍæÔ„š¸Õ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚148.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.6238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NK·Óô—!‚(:» ¨ÍæÔºï­Ø' disk.nvme0c0n1.out‚28662.2626953125) disk./.usagePercent‚74.51372956117768 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0( proc.memory.percent‚17.5534725189209 disk.dm-0.in‚ 194.9453125 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 194.9453125 proc.cpu.threads‚17 network.recv‚ 2.380656e+06 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 194.9453125% disk.nvme0n1.out‚28662.2626953125" d·@Â*Risk.dm-0.out‚28662.2626953125 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 259.83984375 disk.rbd2.out‚190.3544921875$ memory_percent‚91.30425453186035 cpu‚25.381811982785333 disk.rbd3.out‚0% disk./.usageGB‚1110.3084182739258 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 259.83984375 disk.rbd3.in‚0 disk.sda.out‚ 259.83984375 disk.rbd0.in‚0 network.sent‚ 2.0397804e+07 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( proc.memory.availableMB‚ 1424.7109375" proc.memory.rssMB‚ 2875.9609375 disk.sda1.in‚0t¬ê2˜!- _wandbruntime‚6671  _runtime‚6671q:Æ™!À trainer/global_step‚8499 $ _timestamp‚1.788454580131348e+09 _runtime‚6671.685784316 _step‚697 & lr-AdamW/pg1‚4.524418003218342e-05 & lr-AdamW/pg2‚4.524418003218342e-05› ûbËš!Å $ _timestamp‚1.788454580131348e+09 _runtime‚6671.685784316 _step‚697 & lr-AdamW/pg1‚4.524418003218342e-05 & lr-AdamW/pg2‚4.524418003218342e-05 trainer/global_step‚8499¹Ûs]2›!- _wandbruntime‚6671  _runtime‚6671—/½âç'œ!áO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚698 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40755858 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113057822 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232737416192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477613485568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566037 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546038 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190906 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132227072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7477449861632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108282 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548232549852160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 % _timestamp‚1.7884545801321387e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚6671.686786615 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243256320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40756294 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496ñ¡Àéì'!æO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7129132227072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7477449861632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7102227108282 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548232549852160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 % _timestamp‚1.7884545801321387e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚6671.686786615 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502388243256320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40756294 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚698 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40755858 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113057822 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548232737416192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7477613485568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566037 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7102881546038 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123909044 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489367568190906 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944ºì^x´2ž!- _wandbruntime‚6672  _runtime‚6672¦X>ð&Ÿ!êM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495316120430592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295237180416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951668224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113070860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40760726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548295039143936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884545808165002e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7478462512128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482320245213184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 6672.37135469 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123923058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚8499 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090101760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295292416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113070428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40760296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478288415744 _step‚699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0×™ŸN~ !ïM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 86i€àw"8220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489423288756430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495316120430592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548295237180416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951668224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113070860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40760726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548295039143936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884545808165002e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7478462512128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482320245213184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 6672.37135469 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123923058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540816774668288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540816750728192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚8499 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502446090101760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295292416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113070428 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72310134 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40760296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7478288415744 _step‚699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573»ÏD}2¡!- _wandbruntime‚6672  _runtime‚6672•È̾X¢!Ò 2loss/SRqqt2/assignment_loss‚1.0164694819275453 2loss/SRqqt3/assignment_loss‚1.4081707105425925 0loss/SRqqt4/detection_loss‚2.859913458878344 epoch‚7 /loss/FRt3/assignment_loss‚3.318934781683816 /loss/FRt4/assignment_loss‚4.209616632461548 1loss/FBt3/assignment_loss‚0.14707589522004128 1loss/SRqqt2/detection_loss‚1.0279050131422887 /loss/FRt1/detection_loss‚2.4147251293063166 /loss/FRt3/detection_loss‚2.0871401316037885 .loss/FRt4/detection_loss‚3.281365149021149 _runtime‚6672.372815612 2loss/SRqqt1/assignment_loss‚1.0427063220854977 1loss/FBt1/assignment_loss‚0.11681416193459881 1loss/FBt2/assignment_loss‚0.20487222008392983 .loss/FBt4/detection_loss‚2.820480634768804 _step‚700 /loss/FRt1/assignment_loss‚2.016262711584568 -loss/FRt2/detection_loss‚1.70945515489349 /loss/FBt1/detection_loss‚3.5798482298851013 1loss/SRqqt4/assignment_loss‚2.238282383842902 /loss/FBt2/detection_loss‚0.8168866396269157 %loss/total_loss‚44.84718754343247 trainer/global_step‚8499 1loss/FBt4/assignment_loss‚0.14915271025771895 0loss/SRqqt1/detection_loss‚1.567353033813937 1loss/SRqqt3/detection_loss‚1.7032900401230515 % _timestamp‚1.7884545808185194e+09 0loss/FRt2/assignment_loss‚1.5573440982745244 .loss/FBt3/detection_loss‚3.553122818470001ÛOØ]£!× 0loss/FRt2/assignment_loss‚1.5573440982745244 .loss/FBt3/detection_loss‚3.553122818470001 2loss/SRqqt2/assignment_loss‚1.0164694819275453 2loss/SRqqt3/assignment_loss‚1.4081707105425925 0loss/SRqqt4/detection_loss‚2.859913458878344 epoch‚7 /loss/FRt3/assignment_loss‚3.318934781683816 /loss/FRt4/assignment_loss‚4.209616632461548 1loss/FBt3/assignment_loss‚0.14707589522004128 1loss/SRqqt2/detection_loss‚1.0279050131422887 /loss/FRt1/detection_loss‚2.4147251293063166 /loss/FRt3/detection_loss‚2.0871401316037885 .loss/FRt4/detection_loss‚3.281365149021149 _runtime‚6672.372815612 2loss/SRqqt1/assignment_loss‚1.0427063220854977 1loss/FBt1/assignment_loss‚0.11681416193459881 1loss/FBt2/assignment_loss‚0.20487222008392983 .loss/FBt4/detection_loss‚2.820480634768804 _step‚700 /loss/FRt1/assignment_loss‚2.016262711584568 -loss/FRt2/detection_loss‚1.70945515489349 /loss/FBt1/detection_loss‚3.5798482298851013 1loss/SRqqt4/assignment_loss‚2.238282383842902 /loss/FBt2/detection_loss‚0.8168866396269157 %loss/total_loss‚44.84718754343247 trainer/global_step‚8499 1loss/FBt4/assignment_loss‚0.14915271025771895 0loss/SRqqt1/detection_loss‚1.567353033813937 1loss/SRqqt3/detection_loss‚1.7032900401230515 % _timestamp‚1.7884545808185194e+09¼ãt®³¤!‚(:’ ·ÍæÔÅñ•Ö gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.889999999999993 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ¥?¥!‚(:´ ·ÍæÔš¿µØ% disk.nvme0n1.out‚29232.7861328125) disk./.usagePercent‚74.51360846961398 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 2.0402142e+07 disk.drbd1777.out‚0! disk.nvme0c0n1.in‚ 194.9453125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 260.546875$ memory_percent‚91.30167961120605 network.recv‚ 2.383751e+06% disk./.usageGB‚1110.3066139221191 disk.nvme0n1.in‚ 194.9453125 disk.rbd1.out‚ 3.373046875 proc.cpu.threads‚17 disk.sda3.out‚ 260.546875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚29232.7861328125 disk.dm-1.out‚ 260.546875( proc.memory.availableMB‚ 1425.1328125( proc.memory.percent‚17.5534725189209 disk.rbd0.in‚0 disk.dm-0.in‚ 194.9453125 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda1.out‚0 cpu‚25.38103135785464 disk.rbd2.out‚191.9560546875 disk.rbd1.in‚0" disk.dm-0.out‚29232.7861328125 disk.sda2.in‚0" proc.memory.rssMB‚ 2875.9609375¼SßÏž¦!‚(:“ ÆÍæÔÙƒêÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.908999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l\ÕE§!‚(:º ÆÍæÔƶÁØ cpu‚25.380328509488997 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 261.5859375 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17% disk./.usageGB‚1110.3066139221191 disk.dm-1.out‚ 261.5859375 disk.sda.in‚ 0.01171875 disk.sda.out‚ 261.5859375 disk.rbd0.out‚0 network.sent‚ 2.0510878e+07# disk.dm-0.out‚29330.49365234375 disk.rbd1.in‚0 disk.rbd3.out‚0( proc.memory.percent‚17.5534725189209) disk./.usagePercent‚74.51360846961398 disk.rbd1.out‚ 3.373046875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚192.1591796875 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.9453125 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 194.9453125( disk.nvme0c0n1.out‚29330.49365234375 disk.nvme0n1.in‚ 194.9453125 disk.dm-1.in‚ 0.01171875 disk.rbd2.in‚0& disk.nvme0n1.out‚29330.49365234375' proc.memory.availableMB‚ 1424.890625 network.recv‚ 2.389956e+06 disk.sda2.out‚0$ memory_percent‚91.30315780639648ˆÓÿb’¨!‚(:‡ ÕÍæÔï²ìÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚89.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÔ¨D>©!‚(:³ ÕÍæÔû—þ×" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209% disk./.usageGB‚1110.3066711425781 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.out‚193.7568359375) disk./.usagePercent‚74.51361230972701 disk.sda.out‚ 262.03125 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 194.9453125( disk.nvme0c0n1.out‚29330.78662109375 disk.drbd1777.in‚0 disk.dm-0.in‚ 194.9453125 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚25.37959644338308 network.sent‚ 2.0516509e+07 network.recv‚ 2.39428e+06 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚91.30167961120605 disk.rbd3.out‚0 disk.nvme0n1.in‚ 194.9453125( proc.memory.availableMB‚ 1425.1328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 262.03125 disk.sda2.out‚0 disk.dm-1.out‚ 262.03125& disk.nvme0n1.out‚29330.78662109375 disk.drbd1777.out‚0# disk.dm-0.out‚29330.78662109375 disk.sda2.in‚0]°–ý2ª!- _wandbruntime‚6707  _runtime‚6707ýx-ÌÅ«!¿ % _timestamp‚1.7884546160912144e+09 _runtime‚6707.645601899 _step‚701 % lr-AdamW/pg1‚4.41041141303104e-05 % lr-AdamW/pg2‚4.41041141303104e-05 trainer/global_step‚8549gnyʬ!Ä % lr-AdamW/pg2‚4.41041141303104e-05 trainer/global_step‚8549 % _timestamp‚1.7884546160912144e+09 _runtime‚6707.645601899 _step‚701 % lr-AdamW/pg1‚4.41041141303104e-05½¦qüç2­!- _wandbruntime‚6707  _runtime‚6707àñL æ'®!àO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520257111552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058314 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447690 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113690691 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72710366 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8549 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498077437337088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505246881004544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124609844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40980325 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161603085 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124609842 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161583047 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357893985280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚6707.646799031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113690245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4850105646499841èPʯ!åO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72710364 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113690691 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72710366 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124609844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8549 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.³†w!large_pool.allocated‚498077437337088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505246881004544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485010547610624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124609844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40980325 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161603085 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124609842 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161583047 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551357893985280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _runtime‚6707.646799031 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113690245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7520257111552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚492153610058314 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7143062447690¾é4/í2°!- _wandbruntime‚6708  _runtime‚6708ý>ð&±!êM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420597569536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 6708.31737141 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚703 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727849984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40984321 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884546167622921e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161601501 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7521126601728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521286018048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551420414213120 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113702853 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053471232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621565 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113703290 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40984756 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161601501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2xW8õ&²!ïM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551420597569536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 6708.31737141 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚703 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727849984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225359 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161621565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40984321 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884546167622921e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161601501 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7521126601728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7521286018048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551420414213120 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113702853 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053471232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621565 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113703290 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40984756 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161601501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401¿C¶§2³!- _wandbruntime‚6708  _runtime‚67083(F6^´!Ø /loss/FRt3/assignment_loss‚3.221689292540153 2loss/SRqqt4/assignment_loss‚1.5728323668241502 /loss/FBt4/detection_loss‚2.3830397345803003 _runtime‚6708.318901989 0loss/FRt1/assignment_loss‚2.3192827771691715 /loss/FRt4/assignment_loss‚3.463113615910212 2loss/SRqqt3/assignment_loss‚1.3305012244731187 1loss/FBt4/assignment_loss‚0.30500792394344567 /loss/FRt3/detection_loss‚2.5244825780391693 /loss/FRt4/detection_loss‚3.2667758762836456 /loss/FBt1/detection_loss‚2.1116059356265597 2loss/SRqqt2/assignment_loss‚0.6685412321678377 0loss/FBt3/assignment_loss‚0.6989048901945296 1loss/SRqqt4/detection_loss‚2.5040325979391733 trainer/global_step‚8549 2loss/SRqqt1/assignment_loss‚1.4359024044735214 /loss/FBt3/detection_loss‚1.9918835043907166 0loss/FBt1/assignment_loss‚0.3021486795704631 1loss/FBt2/assignment_loss‚0.17758360079631094 /loss/FRt1/detection_loss‚2.3296291194125716 _step‚704 /loss/FRt2/detection_loss‚1.7987053502275019 1loss/SRqqt3/detection_loss‚1.7112967693143422 0loss/SRqqt2/detection_loss‚1.013017464807068 /loss/FBt2/detection_loss‚0.5898814808462105 epoch‚7 0loss/FRt2/assignment_loss‚1.2776916379831276 1loss/SRqqt1/detection_loss‚1.6789411200353732 %loss/total_loss‚40.67649117754868 % _timestamp‚1.7884546167645829e+09~‹àücµ!Ý 0loss/FRt1/assignment_loss‚2.3192827771691715 /loss/FRt4/assignment_loss‚3.463113615910212 2loss/SRqqt3/assignment_loss‚1.3305012244731187 1loss/FBt4/assignment_loss‚0.30500792394344567 /loss/FRt3/detection_loss‚2.5244825780391693 /loss/FRt4/detection_loss‚3.2667758762836456 /loss/FBt1/detection_loss‚2.1116059356265597 2loss/SRqqt2/assignment_loss‚0.6685412321678377 0loss/FBt3/assignment_loss‚0.6989048901945296 1loss/SRqqt4/detection_loss‚2.5040325979391733 trainer/global_step‚8549 2loss/SRqqt1/assignment_loss‚1.4359024044735214 /loss/FBt3/detection_loss‚1.9918835043907166 0loss/FBt1/assignment_loss‚0.3021486795704631 1loss/FBt2/assignment_loss‚0.17758360079631094 /loss/FRt1/detection_loss‚2.3296291194125716 _step‚704 /loss/FRt2/detection_loss‚1.7987053502275019 1loss/SRqqt3/detection_loss‚1.7112967693143422 0loss/SRqqt2/detection_loss‚1.013017464807068 /loss/FBt2/detection_loss‚0.5898814808462105 epoch‚7 0loss/FRt2/assignment_loss‚1.2776916379831276 1loss/SRqqt1/detection_loss‚1.6789411200353732 %loss/total_loss‚40.67649117754868 % _timestamp‚1.7884546167645829e+09 /loss/FRt3/assignment_loss‚3.221689292540153 2loss/SRqqt4/assignment_loss‚1.5728323668241502 /loss/FBt4/detection_loss‚2.3830397345803003 _runtime‚6708.318901989À¾cú“¶!‚(:ˆ äÍæÔÙÁÕÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º±Q>·!‚(:³ äÍæÔ̯ÿ× disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 262.5703125 disk.rbd1.in‚0 network.recv‚ 2.398799e+06% disk./.usageGB‚1110.3066368103027 disk.sda3.out‚ 262.5703125( disk./.usagePercent‚74.5136100056592 disk.rbd2.in‚0& disk.nvme0c0n1.out‚29428.490234375! disk.dm-0.out‚29428.490234375" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209 disk.rbd0.out‚0 disk.dm-0.in‚ 194.9453125 disk.rbd3.in‚0 network.sent‚ 2.0522803e+07! disk.nvme0c0n1.in‚ 194.9453125$ disk.nvme0n1.out‚29428.490234375) proc.memory.availableMB‚ 1424.86328125 disk.rbd3.out‚0 disk.rbd2.out‚195.4091796875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 262.5703125 disk.nvme0n1.in‚ 194.9453125$ memory_percent‚91.30332469940186 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd1.out‚ 3.373046875 cpu‚25.3788634271794}«ž¸!‚(:“ óÍæÔï•çÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.793' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.358600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F‰G¹!‚(:¼ óÍæÔ±©—Ø disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 2.404417e+06( disk.nvme0c0n1.out‚29428.85986328125 disk.nvme0n1.in‚ 194.9453125 disk.sda2.out‚0( proc.memory.percent‚17.5534725189209& disk.nvme0n1.out‚29428.85986328125 disk.dm-0.in‚ 194.9453125% disk./.usageGB‚1110.3058700561523 disk.rbd2.in‚0 disk.sda.out‚ 263.2265625 network.sent‚ 2.0630622e+07 disk.dm-1.out‚ 263.2265625" proc.memory.rssMB‚ 2875.9609375 cpu‚25.378132339082956! disk.nvme0c0n1.in‚ 194.9453125) disk./.usagePercent‚74.51355854814481 disk.drbd1777.in‚0# disk.dm-0.out‚29428.85986328125 disk.sda2.in‚0$ memory_percent‚91.30275249481201) proc.memory.availableMB‚ 1424.95703125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd2.out‚195.4326171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 263.2265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda1.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875ß‚DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125310644 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500895089279488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508105518752768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7563307105280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7884546523037202e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554483480862208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114323001 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114323454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605.%+lì'¿!æO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103838720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7563479117824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41204688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7210447476224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162509099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554483676814848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162509099 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546920197697024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 _step‚706 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125310644 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚500895089279488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508105518752768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7563307105280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7183897787098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7884546523037202e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554483480862208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125310644 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114323001 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114323454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6743.858314856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508106193118208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193118208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41204237Âgç<2À!- _wandbruntime‚6744  _runtime‚6744įoð&Á!êM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852211 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162547617 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222062 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚707 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162527553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554546011113984 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 $ _timestamp‚1.788454652986045e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114336036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365598208 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7564186619392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266904064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41209102 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554546188178944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚487810675308544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508163365598208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660026 trainer/global_step‚8599 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7211266904064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7211923279872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41208670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114335602 _runtime‚6744.540851633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7211266904064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7211923279872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41208670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚500951424315392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114335602 _runtime‚6744.540851633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_trai„ÕŠUn_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚707 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162527553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554546011113984 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 $ _timestamp‚1.788454652986045e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114336036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508163365598208 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7564186619392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7211266904064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41209102 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554546188178944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚487810675308544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ãê~v2Ã!- _wandbruntime‚6744  _runtime‚6744M2pbXÄ!Ò 1loss/SRqqt1/detection_loss‚1.6093916716399017 _runtime‚6744.542394849 ,loss/FBt1/detection_loss‚2.8581180440055 trainer/global_step‚8599 /loss/FRt2/detection_loss‚2.3008789311917055 0loss/FRt3/assignment_loss‚3.2378952584483405 1loss/FBt3/assignment_loss‚0.08455835102358833 0loss/FBt4/assignment_loss‚0.3016448144293718 0loss/SRqqt4/detection_loss‚2.625570543683492 1loss/SRqqt2/detection_loss‚1.0665312767883792 0loss/FRt1/assignment_loss‚3.1556655904080007 2loss/SRqqt1/assignment_loss‚1.1068376454835136 /loss/FRt3/detection_loss‚2.0779062286019623 1loss/SRqqt3/detection_loss‚1.6725985385336966 -loss/FBt4/detection_loss‚2.68792610509055 /loss/FRt4/assignment_loss‚4.084116697311401 2loss/SRqqt4/assignment_loss‚2.2432368913522134 .loss/FRt4/detection_loss‚2.883784406112902 %loss/total_loss‚44.57661818219728 /loss/FRt1/detection_loss‚2.2530361423467067 2loss/FBt1/assignment_loss‚0.042158951232623726 /loss/FBt2/assignment_loss‚0.150070572033368 0loss/FRt2/assignment_loss‚2.3873392386095866 2loss/SRqqt3/assignment_loss‚1.1832170198970247 .loss/FBt2/detection_loss‚0.565800944245746 /loss/FBt3/detection_loss‚3.0028231889009476 2loss/SRqqt2/assignment_loss‚0.9955111308267615 epoch‚7 % _timestamp‚1.7884546529880705e+09 _step‚708Xæ­]Å!× 2loss/FBt1/assignment_loss‚0.042158951232623726 /loss/FBt2/assignment_loss‚0.150070572033368 0loss/FRt2/assignment_loss‚2.3873392386095866 2loss/SRqqt3/assignment_loss‚1.1832170198970247 .loss/FBt2/detection_loss‚0.565800944245746 /loss/FBt3/detection_loss‚3.0028231889009476 2loss/SRqqt2/assignment_loss‚0.9955111308267615 epoch‚7 % _timestamp‚1.7884546529880705e+09 _step‚708 1loss/SRqqt1/detection_loss‚1.6093916716399017 _runtime‚6744.542394849 ,loss/FBt1/detection_loss‚2.8581180440055 trainer/global_step‚8599 /loss/FRt2/detection_loss‚2.3008789311917055 0loss/FRt3/assignment_loss‚3.2378952584483405 1loss/FBt3/assignment_loss‚0.08455835102358833 0loss/FBt4/assignment_loss‚0.3016448144293718 0loss/SRqqt4/detection_loss‚2.625570543683492 1loss/SRqqt2/detection_loss‚1.0665312767883792 0loss/FRt1/assignment_loss‚3.1556655904080007 2loss/SRqqt1/assignment_loss‚1.1068376454835136 /loss/FRt3/detection_loss‚2.0779062286019623 1loss/SRqqt3/detection_loss‚1.6725985385336966 -loss/FBt4/detection_loss‚2.68792610509055 /loss/FRt4/assignment_loss‚4.084116697311401 2loss/SRqqt4/assignment_loss‚2.2432368913522134 .loss/FRt4/detection_loss‚2.883784406112902 %loss/total_loss‚44.57661818219728 /loss/FRt1/detection_loss‚2.2530361423467067Ä·@žÆ!‚(:“ ‚ÎæÔŸ¶¸Õ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›?9Ç!‚(:® ‚ÎæÔÎà¯Ø disk.dm-0.in‚ 194.9453125 cpu‚25.37736757490696 disk.rbd2.out‚197.1474609375& disk.nvme0c0n1.out‚29431.529296875! disk.dm-0.out‚29431.529296875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 263.78125$ memory_percent‚91.30237102508545) disk./.usagePercent‚74.51366939940715 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd3.out‚0 network.recv‚ 2.408717e+06 disk.sda1.out‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 194.9453125 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 263.78125" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17% disk./.usageGB‚1110.3075218200684 disk.sda3.out‚ 263.78125 network.sent‚ 2.063675e+07! disk.nvme0c0n1.in‚ 194.9453125 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd0.out‚0$ disk.nvme0n1.out‚29431.529296875 disk.rbd3.in‚0) proc.memory.availableMB‚ 1425.01953125( proc.memory.percent‚17.5534725189209 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0úöΛÈ!‚(: ‘ÎæÔÕˆ€Ö gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.919999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡NË@É!‚(:µ ‘ÎæÔ÷Ëû×( proc.memory.percent‚17.5534725189209 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 2.0744934e+07 disk.sda1.in‚0 disk.rbd2.out‚197.1748046875! disk.dm-0.out‚29528.982421875 disk.dm-1.out‚ 264.66015625' proc.memory.availableMB‚ 1425.015625$ memory_percent‚91.30239486694336 disk.sda3.out‚ 264.66015625 disk.nvme0n1.in‚ 194.9453125 disk.drbd1777.in‚0$ disk.nvme0n1.out‚29528.982421875 disk.dm-0.in‚ 194.9453125 disk.sda2.in‚0( disk./.usagePercent‚74.5136107736818% disk./.usageGB‚1110.3066482543945 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 network.recv‚ 2.414978e+06 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚29528.982421875 cpu‚25.37668436085071 disk.sda.in‚ 0.01171875 disk.sda.out‚ 264.66015625 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 194.9453125 disk.dm-1.in‚ 0.01171875_¶§Ê!‚(:’  ÎæÔÈÃÑÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.710000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õMJYFË!‚(:»  ÎæÔ…ò§Ø network.sent‚ 2.075117e+07 network.recv‚ 2.419426e+06 disk.dm-0.in‚ 194.9453125 disk.rbd3.out‚0$ memory_percent‚91.30237102508545 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚198.8896484375 disk.drbd1777.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1425.01953125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 194.9453125% disk.nvme0n1.out‚29529.5810546875 disk.sda2.out‚0( proc.memory.percent‚17.5534725189209 proc.cpu.threads‚17 disk.sda.out‚ 265.34765625 disk.sda3.out‚ 265.34765625 disk.rbd1.out‚ 3.373046875% disk./.usageGB‚1110.3075828552246 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875! disk.nvme0c0n1.in‚ 194.9453125' disk.nvme0c0n1.out‚29529.5810546875 disk.rbd3.in‚0 cpu‚25.375924954052962) disk./.usagePercent‚74.51367349552768 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 265.34765625" proc.memory.rssMB‚ 2875.9609375" disk.dm-0.out‚29529.5810546875 disk.rbd1.in‚0w+Í2Ì!- _wandbruntime‚6780  _runtime‚6780¾•Ò‚ÉÍ!à ' lr-AdamW/pg1‚4.1854251014863554e-05 ' lr-AdamW/pg2‚4.1854251014863554e-05 trainer/global_step‚8649 % _timestamp‚1.7884546884554663e+09 _runtime‚6780.009779695 _step‚709uGâÎÎ!È _runtime‚6780.009779695 _step‚709 ' lr-AdamW/pg1‚4.1854251014863554e-05 ' lr-AdamW/pg2‚4.1854251014863554e-05 trainer/global_step‚8649 % _timestamp‚1.7884546884554663e+09Å\TU2Ï!- _wandbruntime‚6780  _runtime‚6780&[Ø*æ'Ð!àO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6780.010807321 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114957916 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503712723218944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609125210112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚497726365270246 _step‚710 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41431186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41430752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606414569984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788454688456154e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433282048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73527164 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503712741221888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830866432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455189 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156500992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830866432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163435151 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550002734580224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435151 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6780.010807321 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114957916 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503712723218944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126011444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609125210112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚497726365270246 _step‚710 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train?º¸_ _batch_start/inactive_split.large_pool.allocated‚73527166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41431186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41430752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7606414569984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788454688456154e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251433282048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928Æüt§³2Ò!- _wandbruntime‚6780  _runtime‚6780N~Îð&Ó!êM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252252709888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚490555881836544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909085696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479063 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114970691 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7607256875008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511021328967680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909085696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557671818386432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003346432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7607433068544 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126025460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884546893681593e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41435799 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚497781414358654 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503769076257792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835978 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999434 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752Ï“ì»õ&Ô!ïM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453605 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41435359 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126025460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73535334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884546893681593e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41435799 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚497781414358654 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503769076257792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚497782085835978 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7226203999434 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252252709888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚490555881836544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909085696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479063 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114970691 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7607256875008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚8649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511021328967680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909085696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126025458 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557671818386432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚511022003346432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7607433068544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794249216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649413512192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126712244 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734503703040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 % _timestamp‚1.7884547248216243e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pooBÆ¿l.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41656775 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553085247523328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7266222903670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41657229 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚493246184233984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381241 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605¯ÝKì'á!æO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7649413512192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126712244 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560734503703040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 % _timestamp‚1.7884547248216243e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41656775 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553085247523328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7266222903670 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41657229 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚493246184233984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381241 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794249216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844ÊÖK2â!- _wandbruntime‚6817  _runtime‚6817Iц«ñ&ã!ëM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126726258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650277993984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8699 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41661187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41661646 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506586746203136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6817.051241824 _step‚715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7267039338842 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115605380 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894891520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560797200195072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650108091904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 % _timestamp‚1.7884547254964793e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379657 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797006352896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115604919+»¼±ä!ðM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6817.051241824 _step‚715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7267039338842 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115605380 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894891520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560797200195072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650108091904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 % _timestamp‚1.7884547254964793e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379657 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚493301071325184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797006352896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115604919 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126726258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650277993984 [MDeviceStatsMonitor.on_train_batch_enÊàOE d/inactive_split_bytes.small_pool.current‚ 169902080 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8699 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41661187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41661646 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506586746203136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928Ë-Šp2å!- _wandbruntime‚6817  _runtime‚68178xõYWæ!Ñ $ _timestamp‚1.788454725498153e+09 _step‚716 2loss/SRqqt2/assignment_loss‚0.9286105899626146 0loss/FBt1/assignment_loss‚0.3012137624929892 /loss/FRt2/detection_loss‚1.7241651824116706 1loss/SRqqt3/detection_loss‚1.8535899587803417 /loss/FBt4/detection_loss‚2.4819988987662573 trainer/global_step‚8699 1loss/FBt2/assignment_loss‚0.13331092165519404 .loss/FRt1/detection_loss‚2.502817067503929 1loss/SRqqt1/assignment_loss‚1.299030905406849 .loss/FRt4/detection_loss‚3.062383742168032 1loss/FBt4/assignment_loss‚0.10475524193250087 .loss/FRt3/detection_loss‚2.304815231689385 /loss/FBt2/detection_loss‚0.5401256627321586 0loss/FRt1/assignment_loss‚2.6435783073306083 1loss/SRqqt1/detection_loss‚1.6885588759300756 /loss/SRqqt2/detection_loss‚0.89915547946441 0loss/SRqqt4/detection_loss‚2.583518067995707 .loss/FBt1/detection_loss‚2.428513157367706 %loss/total_loss‚40.87266640302999 0loss/FRt3/assignment_loss‚2.9181506094478427 1loss/FBt3/assignment_loss‚0.18092050653649494 _runtime‚6817.052458326 /loss/FRt2/assignment_loss‚1.350042338371277 /loss/FRt4/assignment_loss‚3.551860110513095 2loss/SRqqt3/assignment_loss‚1.4076928847365908 /loss/FBt3/detection_loss‚2.3200070038437843 2loss/SRqqt4/assignment_loss‚1.6638518959904711 epoch‚7hïg\ç!Ö /loss/FRt2/assignment_loss‚1.350042338371277 /loss/FRt4/assignment_loss‚3.551860110513095 2loss/SRqqt3/assignment_loss‚1.4076928847365908 /loss/FBt3/detection_loss‚2.3200070038437843 2loss/SRqqt4/assignment_loss‚1.6638518959904711 epoch‚7 $ _timestamp‚1.788454725498153e+09 _step‚716 2loss/SRqqt2/assignment_loss‚0.9286105899626146 0loss/FBt1/assignment_loss‚0.3012137624929892 /loss/FRt2/detection_loss‚1.7241651824116706 1loss/SRqqt3/detection_loss‚1.8535899587803417 /loss/FBt4/detection_loss‚2.4819988987662573 trainer/global_step‚8699 1loss/FBt2/assignment_loss‚0.13331092165519404 .loss/FRt1/detection_loss‚2.502817067503929 1loss/SRqqt1/assignment_loss‚1.299030905406849 .loss/FRt4/detection_loss‚3.062383742168032 1loss/FBt4/assignment_loss‚0.10475524193250087 .loss/FRt3/detection_loss‚2.304815231689385 /loss/FBt2/detection_loss‚0.5401256627321586 0loss/FRt1/assignment_loss‚2.6435783073306083 1loss/SRqqt1/detection_loss‚1.6885588759300756 /loss/SRqqt2/detection_loss‚0.89915547946441 0loss/SRqqt4/detection_loss‚2.583518067995707 .loss/FBt1/detection_loss‚2.428513157367706 %loss/total_loss‚40.87266640302999 0loss/FRt3/assignment_loss‚2.9181506094478427 1loss/FBt3/assignment_loss‚0.18092050653649494 _runtime‚6817.052458326ÌÎ(sû“è!‚(:ˆ ÍÎæÔ€¤ÇÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.499' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.4998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨à–ÏDé!‚(:¹ ÍÎæÔûãÉØ disk.sda1.out‚0" disk.nvme0c0n1.in‚ 194.94921875 disk.drbd1777.in‚0 network.recv‚ 2.4334e+06 disk.rbd0.out‚0! disk.dm-0.out‚29630.388671875 disk.rbd3.in‚0 disk.sda3.out‚ 267.19921875 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.94921875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.0870917e+07 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 267.19921875 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚29630.388671875$ memory_percent‚91.30280017852783) disk./.usagePercent‚74.51379330705372$ disk./.usageGB‚1110.309368133545" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209 cpu‚25.373818499214956 disk.rbd0.in‚0& disk.nvme0c0n1.out‚29630.388671875 disk.nvme0n1.in‚ 194.94921875 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 267.19921875 disk.sda2.in‚0) proc.memory.availableMB‚ 1424.94921875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚200.6552734375€ 91ê!‚(:’ ÜÎæÔœÈòÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.98' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.795999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄcPMë!‚(: ÜÎæÔ‚½ŸØ disk.sda3.in‚ 0.01171875 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.9609375 disk.rbd2.out‚202.3076171875 disk.drbd1777.in‚0# disk.dm-0.out‚29728.43896484375 disk.sda2.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 267.81640625 disk.sda1.in‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚29728.43896484375 disk.drbd1777.out‚0 cpu‚25.373113514143974 disk.rbd3.out‚0 disk.dm-1.out‚ 267.81640625( proc.memory.percent‚17.5534725189209" disk.nvme0c0n1.in‚ 194.94921875) proc.memory.availableMB‚ 1424.43359375 proc.cpu.threads‚17 network.sent‚ 2.0979872e+07 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 network.recv‚ 2.439863e+06% disk./.usageGB‚1110.3078536987305 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚29728.43896484375 disk.nvme0n1.in‚ 194.94921875 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 267.81640625 disk.dm-0.in‚ 194.94921875$ memory_percent‚91.30594730377197) disk./.usagePercent‚74.51369167206262-º2ì!- _wandbruntime‚6852  _runtime‚6852•?šÇí!Á & lr-AdamW/pg1‚3.964630543657181e-05 & lr-AdamW/pg2‚3.964630543657181e-05 trainer/global_step‚8749 % _timestamp‚1.7884547610288947e+09 _runtime‚6852.583335742 _step‚717¾RÌî!Æ & lr-AdamW/pg1‚3.964630543657181e-05 & lr-AdamW/pg2‚3.964630543657181e-05 trainer/global_step‚8749 % _timestamp‚1.7884547610288947e+09 _runtime‚6852.583335742 _step‚717Í{Z#Q2ï!- _wandbruntime‚6852  _runtime‚6852Gβ¤ä'ð!ÞO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚495991373722624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127413044 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74343964 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74343966 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431997440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚556167784406528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7692369018880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333404893696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 trainer/global_step‚8749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7692532642816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516682106362880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚563860153425408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚509348027103744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516682106362880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚563860153425408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚509348027103744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127413044 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74343964 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74343966 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431997440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚556167784406528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7692369018880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088ÎTy°¥2ò!- _wandbruntime‚6853  _runtime‚6853܈²¯ ó!ëM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165325773 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7307220240286 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292752833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚509404380142592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41888440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6853.264558995 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116240572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693226234368 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚516739278842880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334880697344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563922851026432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556229459084288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305709 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516739278842880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563922661378560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556229435144192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127427060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41888893{dìhö&ô!ðM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116240572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7693226234368 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚516739278842880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334880697344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563922851026432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556229459084288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305709 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516739278842880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563922661378560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556229435144192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127427060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41888893 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165325773 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7307220240286 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292752833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚509404380142592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41888440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6853.264558995ϯ¸Ú¿2õ!- _wandbruntime‚6853  _runtime‚6853†ºNVö!Ð %loss/total_loss‚43.72032934906184 0loss/FRt3/assignment_loss‚3.0567567756301477 0loss/SRqqt2/detection_loss‚1.230653627352281 1loss/SRqqt3/detection_loss‚1.7175392319085234 .loss/FBt1/detection_loss‚4.924907386302948 ,loss/FBt4/detection_loss‚2.5181687942573 2loss/SRqqt4/assignment_loss‚1.8560177519774208 /loss/FRt3/detection_loss‚1.9187855846003483 _runtime‚6853.265895924 2loss/SRqqt2/assignment_loss‚1.3968292050740936 1loss/FBt3/assignment_loss‚0.09840681289576671 trainer/global_step‚8749 1loss/FBt2/assignment_loss‚0.16248304597106056 /loss/FRt1/detection_loss‚2.2100358749429385 -loss/FRt4/detection_loss‚3.31110596905152 0loss/SRqqt1/detection_loss‚1.711069135090052 epoch‚7 2loss/SRqqt1/assignment_loss‚0.9381887533749311 .loss/FBt2/detection_loss‚0.793192398291117 .loss/FBt3/detection_loss‚2.181685545227744 0loss/FRt4/assignment_loss‚3.6613020400206246 2loss/SRqqt3/assignment_loss‚1.2093909679580543 0loss/FBt1/assignment_loss‚0.7231506705284119 0loss/FBt4/assignment_loss‚0.1862902432829807 /loss/FRt2/detection_loss‚1.6987747320045645 1loss/SRqqt4/detection_loss‚2.4685737089468884 % _timestamp‚1.7884547617114918e+09 0loss/FRt2/assignment_loss‚1.6978204628595939 _step‚720 /loss/FRt1/assignment_loss‚2.0492006315125368n'½[÷!Õ /loss/FRt1/assignment_loss‚2.049200631512536 %loss/total_loss‚43.72032934906184 0loss/FRt3/assignment_loss‚3.0567567756301477 0loss/SRqqt2/detection_loss‚1.230653627352281 1loss/SRqqt3/detection_loss‚1.7175392319085234 .loss/FBt1/detection_loss‚4.924907386302948 ,loss/FBt4/detection_loss‚2.5181687942573 2loss/SRqqt4/assignment_loss‚1.8560177519774208 /loss/FRt3/detection_loss‚1.9187855846003483 _runtime‚6853.265895924 2loss/SRqqt2/assignment_loss‚1.3968292050740936 1loss/FBt3/assignment_loss‚0.09840681289576671 trainer/global_step‚8749 1loss/FBt2/assignment_loss‚0.16248304597106056 /loss/FRt1/detection_loss‚2.2100358749429385 -loss/FRt4/detection_loss‚3.31110596905152 0loss/SRqqt1/detection_loss‚1.711069135090052 epoch‚7 2loss/SRqqt1/assignment_loss‚0.9381887533749311 .loss/FBt2/detection_loss‚0.793192398291117 .loss/FBt3/detection_loss‚2.181685545227744 0loss/FRt4/assignment_loss‚3.6613020400206246 2loss/SRqqt3/assignment_loss‚1.2093909679580543 0loss/FBt1/assignment_loss‚0.7231506705284119 0loss/FBt4/assignment_loss‚0.1862902432829807 /loss/FRt2/detection_loss‚1.6987747320045645 1loss/SRqqt4/detection_loss‚2.4685737089468884 % _timestamp‚1.7884547617114918e+09 0loss/FRt2/assignment_loss‚1.6978204628595939 _step‚720ЈÂQO“ø!‚(:ˆ ëÎæÔ¼…ÀÕ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e‰ÊãJù!‚(:¿ ëÎæÔ’‡Ù# disk.dm-0.out‚29728.75146484375) disk./.usagePercent‚74.51382863609344 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1424.89453125 disk.nvme0n1.in‚ 194.94921875 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2875.9609375 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875( proc.memory.percent‚17.5534725189209 proc.cpu.threads‚17& disk.nvme0n1.out‚29728.75146484375 disk.sda2.in‚0 disk.dm-1.out‚ 268.4921875 disk.sda1.in‚0 network.sent‚ 2.0985221e+07( disk.nvme0c0n1.out‚29728.75146484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0 disk.rbd2.out‚203.9052734375 network.recv‚ 2.443644e+06" disk.nvme0c0n1.in‚ 194.94921875 disk.dm-0.in‚ 194.94921875 disk.sda.out‚ 268.4921875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 268.4921875% disk./.usageGB‚1110.3098945617676 disk.rbd0.out‚0$ memory_percent‚91.30313396453857 cpu‚25.372403015581277tºØú!‚(:’ úÎæÔ°¤ÅÕ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚96.022' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.204400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û+)ðJû!‚(:¿ úÎæÔ•Ä±× network.sent‚ 2.1094227e+07 disk.rbd1.in‚0 disk.sda.out‚ 268.95703125 disk.drbd1777.in‚0 disk.sda2.out‚0) disk./.usagePercent‚74.51369576818317 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0! disk.rbd2.out‚204.08154296875 disk.dm-1.in‚ 0.01171875 cpu‚25.371678139176616 network.recv‚ 2.450126e+06 disk.rbd2.in‚0" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 268.95703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 194.94921875 disk.rbd3.in‚0$ memory_percent‚94.19801235198975 disk.dm-1.out‚ 268.95703125( proc.memory.percent‚17.5534725189209% disk./.usageGB‚1110.3079147338867 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 194.94921875 disk.drbd1777.out‚0" disk.dm-0.out‚29826.3388671875 disk.sda2.in‚0' disk.nvme0c0n1.out‚29826.3388671875% disk.nvme0n1.out‚29826.3388671875 disk.dm-0.in‚ 194.94921875( proc.memory.availableMB‚ 950.59765625Ucír“ü!‚(:ˆ ‰ÏæÔåÄÜÕ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ô¶kIý!‚(:¾ ‰ÏæÔ®Ë¯Ø disk.sda1.out‚0" proc.memory.rssMB‚ 2875.9609375 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚29923.99951171875 disk.dm-0.in‚ 194.94921875 disk.drbd1777.in‚0$ memory_percent‚94.19236183166504 network.recv‚ 2.454537e+06) disk./.usagePercent‚74.51370754452975 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 269.63671875 disk.rbd2.in‚0# disk.dm-0.out‚29923.99951171875 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 269.63671875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 194.94921875( disk.nvme0c0n1.out‚29923.99951171875 disk.nvme0n1.in‚ 194.94921875' proc.memory.availableMB‚ 951.5234375$ disk./.usageGB‚1110.308090209961 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0( proc.memory.percent‚17.5534725189209 proc.cpu.threads‚17 network.sent‚ 2.10998e+07 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0 cpu‚25.370972381350768 disk.rbd3.in‚0 disk.sda.out‚ 269.63671875! disk.rbd2.out‚205.62060546875+{ì2þ!- _wandbruntime‚6888  _runtime‚6888C·u“Æÿ!À & lr-AdamW/pg1‚3.855862826659116e-05 & lr-AdamW/pg2‚3.855862826659116e-05 trainer/global_step‚8799 $ _timestamp‚1.788454797103051e+09 _runtime‚6888.657480322 _step‚721ê-Ë€"Å & lr-AdamW/pg1‚3.855862826659116e-05 & lr-AdamW/pg2‚3.855862826659116e-05 trainer/global_step‚8799 $ _timestamp‚1.788454797103051e+09 _runtime‚6888.657480322 _step‚721Ñ[ùl2"- _wandbruntime‚6888  _runtime‚6888s-à‚"áO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566985794700288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884547971038315e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7375047062016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166213307 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116864467 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚8799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233345 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512165679046144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519540069745664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_sj]µ"tart/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512165697049088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128113844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚6888.658541632 _step‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42111651 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566985611330560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047062016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166233345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166213307 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116864467 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚8799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166233345 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512165679046144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519540069745664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512165697049088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128113844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚6888.658541632 _step‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42111651 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566985611330560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047062016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166233345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884547971038315e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7375047062016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908Ò2¡2„"- _wandbruntime‚6889  _runtime‚6889p˜ßiñ&…"ëM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166251825 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567048115050496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512222050087936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚723 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210127360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116876730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚8799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242212352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116198 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2„q*ö&†"ðM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166251825 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567048115050496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512222050087936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498791501420544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚723 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7375210127360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116876730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74760534 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚8799 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242212352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116198 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75160766 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778528253952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501481786778624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42340605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778370921472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570111410366976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884548332110794e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514983330988544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032867840 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159397 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388074484138 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522398707493888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 _step‚726 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111229094400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117500906 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167159397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522398707493888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 _step‚726 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570111229094400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117500906 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167159397 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75160766 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7778528253952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501481786778624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42340605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778370921472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570111410366976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ÖÜA>Î2”"- _wandbruntime‚6925  _runtime‚6925—·¶ôÕ•"ëM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501536690909184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879960576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117513609 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42344677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157813 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501536707948544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 % _timestamp‚1.7884548338851707e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852308992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515039684027392 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167177877 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚515039702030336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305610 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117514071 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42345137 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 EDeviceStatsMonitor.on_train_batch_end/< cêallocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195933184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522456554339328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295986471 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚515039702030336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508926253305610 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117514071 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42345137 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195933184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522456554339328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295986471 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879960576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117513609 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42344677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167157813 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501536707948544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 % _timestamp‚1.7884548338851707e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7416852308992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515039684027392 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944ר6§„2—"- _wandbruntime‚6925  _runtime‚6925*¸Œ€]˜"× /loss/FRt2/detection_loss‚1.7666364175788427 /loss/FBt2/detection_loss‚0.6536530452183609 trainer/global_step‚8849 0loss/FBt3/assignment_loss‚0.7060407144017518 1loss/SRqqt4/detection_loss‚2.4057685243231908 .loss/FBt1/detection_loss‚2.744575226306915 % _timestamp‚1.7884548338872876e+09 2loss/SRqqt1/assignment_loss‚1.7043575064363805 2loss/SRqqt4/assignment_loss‚1.4126663878560066 /loss/SRqqt1/detection_loss‚1.65904276445508 1loss/SRqqt2/detection_loss‚0.8542256442376891 /loss/FBt4/detection_loss‚2.7864293796675548 %loss/total_loss‚42.91097993001764 1loss/FBt2/assignment_loss‚0.16389405207482846 /loss/FRt3/detection_loss‚1.9938327030080263 1loss/SRqqt3/detection_loss‚1.9124965276569128 epoch‚7 _runtime‚6925.441635748 0loss/FRt1/assignment_loss‚2.8522780179977416 /loss/FRt4/detection_loss‚3.1219479830368706 0loss/FRt2/assignment_loss‚1.2600083260212915 0loss/FRt4/assignment_loss‚3.6417115045630415 0loss/FBt1/assignment_loss‚0.5637046087300405 2loss/SRqqt3/assignment_loss‚1.2892570112831891 /loss/FRt3/assignment_loss‚2.822530357564082 2loss/SRqqt2/assignment_loss‚1.0864920569835483 /loss/FRt1/detection_loss‚2.2795728555321695 /loss/FBt3/detection_loss‚2.9391343742609024 _step‚728 1loss/FBt4/assignment_loss‚0.29072394082322717;e•b™"Ü /loss/FRt4/detection_loss‚3.1219479830368706 0loss/FRt2/assignment_loss‚1.2600083260212915 0loss/FRt4/assignment_loss‚3.6417115045630415 0loss/FBt1/assignment_loss‚0.5637046087300405 2loss/SRqqt3/assignment_loss‚1.2892570112831891 /loss/FRt3/assignment_loss‚2.822530357564082 2loss/SRqqt2/assignment_loss‚1.0864920569835483 /loss/FRt1/detection_loss‚2.2795728555321695 /loss/FBt3/detection_loss‚2.9391343742609024 _step‚728 1loss/FBt4/assignment_loss‚0.29072394082322717 /loss/FRt2/detection_loss‚1.7666364175788427 /loss/FBt2/detection_loss‚0.6536530452183609 trainer/global_step‚8849 0loss/FBt3/assignment_loss‚0.7060407144017518 1loss/SRqqt4/detection_loss‚2.4057685243231908 .loss/FBt1/detection_loss‚2.744575226306915 % _timestamp‚1.7884548338872876e+09 2loss/SRqqt1/assignment_loss‚1.7043575064363805 2loss/SRqqt4/assignment_loss‚1.4126663878560066 /loss/SRqqt1/detection_loss‚1.65904276445508 1loss/SRqqt2/detection_loss‚0.8542256442376891 /loss/FBt4/detection_loss‚2.7864293796675548 %loss/total_loss‚42.91097993001764 1loss/FBt2/assignment_loss‚0.16389405207482846 /loss/FRt3/detection_loss‚1.9938327030080263 1loss/SRqqt3/detection_loss‚1.9124965276569128 epoch‚7 _runtime‚6925.441635748 0loss/FRt1/assignment_loss‚2.8522780179977416Ø…¡Â&žš"‚(:“ ¶ÏæÔÆÖÕ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.858400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎùW4F›"‚(:» ¶ÏæÔ·ì—Ù network.recv‚ 2.469023e+06% disk./.usageGB‚1110.3100395202637 disk.rbd2.in‚0 disk.nvme0n1.in‚ 194.94921875# disk.dm-0.out‚30022.64208984375 disk.sda2.in‚0 disk.sda.out‚ 271.171875) disk./.usagePercent‚74.51383836437975 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.9609375 cpu‚25.368889716172234& disk.nvme0n1.out‚30022.64208984375 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 271.171875 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 0.01171875( proc.memory.percent‚17.5534725189209 disk.sda1.in‚0 disk.sda1.out‚0! disk.rbd2.out‚207.45263671875 disk.dm-0.in‚ 194.94921875 disk.rbd0.in‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 271.171875 network.sent‚ 2.1220149e+07" disk.nvme0c0n1.in‚ 194.94921875( disk.nvme0c0n1.out‚30022.64208984375 disk.rbd3.out‚0 disk.drbd1777.out‚0$ memory_percent‚94.19541358947754' proc.memory.availableMB‚ 951.0234375 disk.drbd1777.in‚08²Ì‘œ"‚(:† ÅÏæÔú­ÉÕ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»—ZSJ"‚(:¿ ÅÏæÔ°ÝÙ network.recv‚ 2.475241e+06 disk.dm-0.in‚ 194.94921875# disk.dm-0.out‚30120.67919921875 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17% disk./.usageGB‚1110.3100776672363& disk.nvme0n1.out‚30120.67919921875 disk.dm-1.out‚ 271.77734375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 271.77734375! disk.rbd2.out‚208.99169921875" disk.nvme0c0n1.in‚ 194.94921875 disk.nvme0n1.in‚ 194.94921875 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 network.sent‚ 2.1328283e+07 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 271.77734375" proc.memory.rssMB‚ 2875.9609375 disk.drbd1777.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 951.4609375 cpu‚25.36819937723584) disk./.usagePercent‚74.51384092445508 disk.sda2.in‚0 disk.rbd0.out‚0( proc.memory.percent‚17.5534725189209 disk.rbd2.in‚0( disk.nvme0c0n1.out‚30120.67919921875 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 0.01171875# memory_percent‚94.1927433013916ÈHö…œž"‚(:‘ ÔÏæÔ”±ÊÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.339999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áºä\LŸ"‚(:Á ÔÏæÔˆ—±Ø disk.sda2.out‚0 disk.rbd1.out‚ 3.521484375% disk./.usageGB‚1110.3100242614746 disk.rbd2.in‚0! disk.rbd2.out‚210.58154296875 disk.dm-0.in‚ 194.94921875 disk.rbd0.in‚0 disk.dm-1.out‚ 272.29296875 disk.sda1.in‚0 disk.sda3.out‚ 272.29296875" disk.nvme0c0n1.in‚ 194.94921875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0# disk.dm-0.out‚30218.22216796875 disk.sda.in‚ 0.01171875$ memory_percent‚94.19424533843994( proc.memory.availableMB‚ 951.21484375" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 network.sent‚ 2.1334536e+07 disk.rbd0.out‚0( proc.memory.percent‚17.5534725189209) disk./.usagePercent‚74.51383734034961( disk.nvme0c0n1.out‚30218.22216796875 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 cpu‚25.36752621711505 disk.rbd3.in‚0 disk.sda.out‚ 272.29296875 network.recv‚ 2.479503e+06 disk.nvme0n1.in‚ 194.94921875& disk.nvme0n1.out‚30218.22216796875 disk.drbd1777.in‚0ºp Á2 "- _wandbruntime‚6960  _runtime‚6960¿/:È¡" ' lr-AdamW/pg1‚3.6416998469182475e-05 ' lr-AdamW/pg2‚3.6416998469182475e-05 trainer/global_step‚8899 $ _timestamp‚1.788454869173287e+09 _runtime‚6960.727667714 _step‚729a±.-Í¢"Ç ' lr-AdamW/pg2‚3.6416998469182475e-05 trainer/global_step‚8899 $ _timestamp‚1.788454869173287e+09 _runtime‚6960.727667714 _step‚729 ' lr-AdamW/pg1‚3.6416998469182475e-05Ù΋2£"- _wandbruntime‚6960  _runtime‚6960ƒiœàç'¤"áO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019607552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚730 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565415395056128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 % _timestamp‚1.7884548691741967e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6960.728937396 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823546 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚517801000933888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525258019607552 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362311168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118136744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573236888132096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42567124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600893 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168065411 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573237081987584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345242112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7821662991360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118136288 trainer/global_step‚8899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 101942558726 Îlÿ¥"æO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42567578 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168065411 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57323708qý5í#1987584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345242112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7821662991360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118136288 trainer/global_step‚8899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019607552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚730 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565415395056128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 % _timestamp‚1.7884548691741967e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6960.728937396 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823546 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚517801000933888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525258019607552 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362311168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118136744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573236888132096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42567124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600893 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42572024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168083865 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚573299592202752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6961.400015351 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525315192087552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7430380696474 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118149358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7457838114816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517857353972736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181739008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129529458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192087552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517708800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129529458 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7822369344000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚517857335969792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42571587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838114816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299415137792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118148919ÍÚÕñö&¨"ðM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7457838114816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚573299415137792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118148919 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚731 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083865 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7822522468864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884548698451307e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42572024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168083865 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚573299592202752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6961.400015351 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525315192087552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7430380696474 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118149358 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7457838114816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚517857353972736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7457181739008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129529458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192087552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525314517708800 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129529458 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7822369344000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚517857335969792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103929 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42571587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844Ûg9Öý2©"- _wandbruntime‚6961  _runtime‚6961‹˜¤pZª"Ô /loss/FRt3/detection_loss‚2.1353106644419886 % _timestamp‚1.7884548698472428e+09 1loss/FBt1/assignment_loss‚0.19926328498903395 1loss/SRqqt1/detection_loss‚1.7273710686333326 1loss/SRqqt4/detection_loss‚2.8545018258419903 -loss/FBt1/detection_loss‚2.57659626858575 /loss/FBt4/detection_loss‚2.4374283918967614 /loss/FRt3/assignment_loss‚2.923931214544508 /loss/FBt3/assignment_loss‚0.231831641603776 .loss/FRt4/detection_loss‚3.454897165298462 1loss/SRqqt2/detection_loss‚1.0771333905337732 0loss/FRt2/assignment_loss‚1.5648263005109935 2loss/SRqqt2/assignment_loss‚1.0533088198804026 /loss/FBt2/detection_loss‚0.6113122897900621 /loss/FBt3/detection_loss‚2.5410119543472924 epoch‚7 _runtime‚6961.401567145 0loss/FRt1/assignment_loss‚2.6094296878238894 0loss/FBt4/assignment_loss‚0.0798514794307546 .loss/FRt2/detection_loss‚1.835046092191568 /loss/FRt1/detection_loss‚2.0405232927428103 %loss/total_loss‚42.38882584769837 _step‚732 0loss/FRt4/assignment_loss‚3.4677190553574335 2loss/SRqqt3/assignment_loss‚1.7096163487814842 1loss/SRqqt4/assignment_loss‚1.901555716652762 1loss/FBt2/assignment_loss‚0.11363812719950007 1loss/SRqqt3/detection_loss‚1.9272919373309358 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.3154298292891102x-YY_«"Ù /loss/FBt2/detection_loss‚0.6113122897900621 /loss/FBt3/detection_loss‚2.5410119543472924 epoch‚7 _runtime‚6961.401567145 0loss/FRt1/assignment_loss‚2.6094296878238894 0loss/FBt4/assignment_loss‚0.0798514794307546 .loss/FRt2/detection_loss‚1.835046092191568 /loss/FRt1/detection_loss‚2.0405232927428103 %loss/total_loss‚42.38882584769837 _step‚732 0loss/FRt4/assignment_loss‚3.4677190553574335 2loss/SRqqt3/assignment_loss‚1.7096163487814842 1loss/SRqqt4/assignment_loss‚1.901555716652762 1loss/FBt2/assignment_loss‚0.11363812719950007 1loss/SRqqt3/detection_loss‚1.9272919373309358 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.3154298292891102 /loss/FRt3/detection_loss‚2.1353106644419886 % _timestamp‚1.7884548698472428e+09 1loss/FBt1/assignment_loss‚0.19926328498903395 1loss/SRqqt1/detection_loss‚1.7273710686333326 1loss/SRqqt4/detection_loss‚2.8545018258419903 -loss/FBt1/detection_loss‚2.57659626858575 /loss/FBt4/detection_loss‚2.4374283918967614 /loss/FRt3/assignment_loss‚2.923931214544508 /loss/FBt3/assignment_loss‚0.231831641603776 .loss/FRt4/detection_loss‚3.454897165298462 1loss/SRqqt2/detection_loss‚1.0771333905337732 0loss/FRt2/assignment_loss‚1.5648263005109935 2loss/SRqqt2/assignment_loss‚1.0533088198804026ÜïÝvž¬"‚(:“ ãÏæÔÐëæÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.861' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.172199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-YQ2­"‚(:¿ ãÏæÔϦÁØ disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 194.94921875( disk.nvme0c0n1.out‚30218.81591796875& disk.nvme0n1.out‚30218.81591796875 disk.dm-1.out‚ 272.6953125 disk.rbd1.out‚ 3.521484375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda7ÂðÔ1.out‚0 disk.nvme0n1.in‚ 194.94921875 disk.rbd3.out‚0$ memory_percent‚94.19729709625244 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 network.sent‚ 2.1339897e+07% disk./.usageGB‚1110.3101654052734 disk.drbd1777.out‚0( proc.memory.availableMB‚ 950.71484375" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 network.recv‚ 2.483384e+06) disk./.usagePercent‚74.51384681262837 disk.rbd0.out‚0 disk.dm-0.in‚ 194.94921875 disk.sda2.out‚0 disk.sda.out‚ 272.6953125 cpu‚25.366809192925434! disk.rbd2.out‚210.60498046875# disk.dm-0.out‚30218.81591796875 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875( proc.memory.percent‚17.5534725189209 disk.sda3.out‚ 272.6953125UÔG®"‚(:… òÏæÔ ÞºÖ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚93.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àmî‚C¯"‚(:¸ òÏæÔþ±ÅÙ disk.sda3.in‚ 0.01171875 network.sent‚ 2.1448697e+07 network.recv‚ 2.489832e+06% disk./.usageGB‚1110.3101501464844 disk.sda1.in‚0( proc.memory.availableMB‚ 950.67578125 disk.nvme0n1.in‚ 194.94921875 disk.drbd1777.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚74.51384578859823 disk.rbd3.out‚0& disk.nvme0n1.out‚30316.33935546875 disk.dm-0.in‚ 194.94921875 disk.sda2.in‚0 disk.dm-1.out‚ 273.265625 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda.out‚ 273.265625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 194.94921875# disk.dm-0.out‚30316.33935546875$ memory_percent‚94.19753551483154( proc.memory.percent‚17.5534725189209 disk.rbd1.out‚ 3.5537109375 disk.rbd3.in‚0 disk.rbd2.in‚0 cpu‚25.366088425023865 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda3.out‚ 273.265625( disk.nvme0c0n1.out‚30316.33935546875" proc.memory.rssMB‚ 2875.9609375 disk.sda1.out‚0 disk.rbd2.out‚ 212.265625Æ{ÿ2°"- _wandbruntime‚6996  _runtime‚6996»6õ9ɱ"à _runtime‚6996.857248156 _step‚733 ' lr-AdamW/pg1‚3.5363492154194236e-05 ' lr-AdamW/pg2‚3.5363492154194236e-05 trainer/global_step‚8949 % _timestamp‚1.7884549053028657e+096ú$3β"È ' lr-AdamW/pg2‚3.5363492154194236e-05 trainer/global_step‚8949 % _timestamp‚1.7884549053028657e+09 _runtime‚6996.857248156 _step‚733 ' lr-AdamW/pg1‚3.5363492154194236e-05ݲpâµ2³"- _wandbruntime‚6996  _runtime‚6996î`*<ç'´"áO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162954 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130216242 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004479488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚528115982990336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118772312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576362642843648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227745 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576362446891008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6996.858305738 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169011501 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474694 trainer/global_step‚8949 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚528116657355776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991463 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚734 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520618634873344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42794292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 % _timestamp‚1.7884549053036404e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚528115982990336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118772312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576362642843648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568497955879424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299227745 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576362446891008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚6996.858305738 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169011501 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474694 trainer/global_step‚8949 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚528116657355776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991463 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚734 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520618634873344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7470399600710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42794292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 % _timestamp‚1.7884549053036404e+09 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚528173155457024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568559606617088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8949 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520674987912192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118784594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42799309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚6997.532546559 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507027103965184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425147844608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7498167544832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130230258 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118785043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7865379714048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7498823920640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568559582676992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118784594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42799309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚6997.532546559 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507027103965184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576425147844608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7498167544832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130230258 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299260241 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118785043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7865379714048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7498823920640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568559582676992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872ßN¤Ã2¹"- _wandbruntime‚6997  _runtime‚6997éÇÅ)Yº"Ó /loss/FRt1/assignment_loss‚3.096397571620487 0loss/FRt2/assignment_loss‚1.3972960391214915 0loss/FRt3/assignment_loss‚3.2346548517545064 1loss/FBt1/assignment_loss‚0.08638661380973645 0loss/SRqqt3/detection_loss‚1.596589807355613 _step‚736 2loss/SRqqt1/assignment_loss‚1.3659100072844974 1loss/FBt2/assignment_loss‚0.15912971819016725 0loss/FBt3/assignment_loss‚0.1419117613695562 0loss/FBt4/assignment_loss‚0.4310353267355822 .loss/FRt3/detection_loss‚2.037230368455274 1loss/SRqqt1/detection_loss‚1.6702913459179536 &loss/total_loss‚45.395042682898904 % _timestamp‚1.7884549059796543e+09 /loss/FRt4/assignment_loss‚4.147051219586973 2loss/SRqqt4/assignment_loss‚1.9030701126903296 .loss/FRt4/detection_loss‚3.082910599531951 2loss/SRqqt2/assignment_loss‚1.3148245639873273 1loss/SRqqt3/assignment_loss‚1.519446011501969 1loss/SRqqt2/detection_loss‚0.9306617121804844 epoch‚7 /loss/FRt1/detection_loss‚2.4516721790035567 /loss/FBt1/detection_loss‚3.5573525547981264 .loss/FBt3/detection_loss‚4.432908803224564 trainer/global_step‚8949 _runtime‚6997.533945636 /loss/FRt2/detection_loss‚1.7030853724905424 0loss/SRqqt4/detection_loss‚2.171778804063797 /loss/FBt2/detection_loss‚0.7320709137926183 .loss/FBt4/detection_loss‚2.231376424431801Œ!]<^»"Ø /loss/FRt1/assignment_loss‚3.096397571620487 0loss/FRt2/assignment_loss‚1.3972960391214915 0loss/FRt3/assignment_loss‚3.2346548517545064 1loss/FBt1/assignment_loss‚0.08638661380973645 0loss/SRqqt3/detection_loss‚1.596589807355613 _step‚736 2loss/SRqqt1/assignment_loss‚1.3659100072844974 1loss/FBt2/assignment_loss‚0.15912971819016725 0loss/FBt3/assignment_loss‚0.1419117613695562 0loss/FBt4/assignment_loss‚0.4310353267355822 .loss/FRt3/detection_loss‚2.037230368455274 1loss/SRqqt1/detection_loss‚1.6702913459179536 &loss/total_loss‚45.395042682898904 % _timestamp‚1.7884549059796543e+09 /loss/FRt4/assignment_loss‚4.147051219586973 2loss/SRqqt4/assignment_loss‚1.9030701126903296 .loss/FRt4/detection_loss‚3.082910599531951 2loss/SRqqt2/assignment_loss‚1.3148245639873273 1loss/SRqqt3/assignment_loss‚1.519446011501969 1loss/SRqqt2/detection_loss‚0.9306617121804844 epoch‚7 /loss/FRt1/detection_loss‚2.4516721790035567 /loss/FBt1/detection_loss‚3.5573525547981264 .loss/FBt3/detection_loss‚4.432908803224564 trainer/global_step‚8949 _runtime‚6997.533945636 /loss/FRt2/detection_loss‚1.7030853724905424 0loss/SRqqt4/detection_loss‚2.171778804063797 /loss/FBt2/detection_loss‚0.7320709137926183 .loss/FBt4/detection_loss‚2.231376424431801àF—óù¼"‚(:’ ÐæÔ˜ÕÎÕ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.52' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.104000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Ì£F½"‚(:» ÐæÔσŽÙ disk.nvme0n1.in‚ 194.94921875 disk.drbd1777.in‚0# disk.dm-0.out‚30316.94873046875" proc.memory.rssMB‚ 2875.9609375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 194.94921875 disk.dm-1.out‚ 274.0390625 disk.sda1.out‚0 disk.sda.in‚ 0.01171875$ memory_percent‚94.19491291046143% disk./.usageGB‚1110.3101501464844 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 274.0390625 network.recv‚ 2.49333e+06 disk.rbd0.out‚0 disk.rbd2.out‚ 212.29296875( disk.nvme0c0n1.out‚30316.94873046875& disk.nvme0n1.out‚30316.94873046875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 274.0390625 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 3.5537109375 disk.sda1.in‚0 network.sent‚ 2.1453281e+07" disk.nvme0c0n1.in‚ 194.94921875 disk.drbd1777.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 951.10546875( proc.memory.percent‚17.5534725189209 cpu‚25.36541735576866) disk./.usagePercent‚74.51384578859823êŒ9’¾"‚(:‡ ÐæÔ½ÈÕÕ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ViV÷A¿"‚(:¶ ÐæÔ¤³÷Ø disk.rbd1.out‚ 3.5537109375 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚30414.44091796875 disk.nvme0n1.in‚ 194.94921875 disk.dm-0.in‚ 194.94921875 disk.sda2.out‚0 cpu‚25.364711713489214 disk.dm-1.out‚ 274.859375" disk.nvme0c0n1.in‚ 194.94921875 disk.drbd1777.in‚0& proc.memory.availableMB‚ 951.109375 disk.rbd1.in‚0 disk.rbd2.in‚0# disk.dm-0.out‚30414.44091796875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 274.859375& disk.nvme0n1.out‚30414.44091796875" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 disk.drbd1777.out‚0 network.recv‚ 2.499897e+06% disk./.usageGB‚1110.3101501464844 disk.dm-1.in‚ 0.01171875 disk.rbd2.out‚ 213.953125$ memory_percent‚94.19488906860352( proc.memory.percent‚17.5534725189209 network.sent‚ 2.1562303e+07) disk./.usagePercent‚74.51384578859823 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 274.859375 disk.sda1.in‚0%Šr2À"- _wandbruntime‚7032  _runtime‚7032xNÇÁ"Á & lr-AdamW/pg1‚3.432181843391322e-05 & lr-AdamW/pg2‚3.432181843391322e-05 trainer/global_step‚8999 % _timestamp‚1.7884549411972106e+09 _runtime‚7032.751652967 _step‚737d\ÌÂ"Æ % _timestamp‚1.7884549411972106e+09 _runtime‚7032.751652967 _step‚737 & lr-AdamW/pg1‚3.432181843391322e-05 & lr-AdamW/pg2‚3.432181843391322e-05 trainer/global_step‚8999á|R82Ã"- _wandbruntime‚7032  _runtime‚7032³¾¯å'Ä"ßO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130917042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523436286815744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917515 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43021777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚738 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7907608283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333922816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488101046272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119408166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571580468822528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509717423401984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854597 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834557 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579487921870848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523436304818688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119407741 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523436286815744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ì0\¨ÿÅ"äO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130917042 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523436286815744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917515 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43021777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚738 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7907608283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333922816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488101046272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119408166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571580468822528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509717423401984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854597 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834557 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579487921870848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523436304818688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119407741 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523436286815744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952â0ÀØõ2Æ"- _wandbruntime‚7033  _runtime‚7033×ä'Jñ&Ç"ëM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579550446508032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509772327532544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884549418818128e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956033 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43026764 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚739 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119420465 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908482169856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43026333 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7511396937326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7033.436886597 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969C ÈMö&È"ðM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523492639854592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130931058 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7033.436886597 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571642143500288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579550446508032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509772327532544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884549418818128e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956033 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43026764 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚739 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119420465 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509772310493184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7908482169856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43026333 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7511396937326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533833932852224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841770 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43250535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚7069.029502336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43250097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7950663885824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279526 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582613693531648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884549774748614e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526253956761088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613505967616 trainer/global_step‚9049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932852224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843567 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7579976091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7579319728640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7579976091136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7579319728640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533833932852224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841770 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43250535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚7069.029502336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526253938758144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43250097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7950663885824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279526 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526253938758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582613693531648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884549774748614e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526253956761088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582613505967616 trainer/global_step‚9049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932852224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170843567æû¥&2Ø"- _wandbruntime‚7069  _runtime‚7069ZçëÖð&Ù"êM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139156480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520070420775242 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862021 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76802532 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚9049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170882085 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788454978147232e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533891105332224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚533891105332224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493879 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795532288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7069.702054724 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120057109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43255005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276734 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120057539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951368447488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676024890880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131631860 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7951529960960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795532288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676210344448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882085 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676024890880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131631860 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7951529960960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7580795532288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676210344448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882085 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7580795532288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7069.702054724 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120057109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43255005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276734 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120057539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951368447488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424çÒ¿T2Û"- _wandbruntime‚7069  _runtime‚7069YtÇN[Ü"Õ .loss/FRt2/assignment_loss‚1.82472832247896 0loss/FRt3/assignment_loss‚3.5410339660114714 .loss/FRt3/detection_loss‚2.378612259361479 1loss/SRqqt2/detection_loss‚0.9705043812073878 .loss/FBt4/detection_loss‚2.346447695385326 0loss/FRt4/assignment_loss‚3.1160980979601542 /loss/FRt2/detection_loss‚1.6979104189096756 % _timestamp‚1.7884549781493049e+09 1loss/FBt4/assignment_loss‚0.35034888481128623 2loss/SRqqt2/assignment_loss‚1.0133998342164194 1loss/FBt3/assignment_loss‚0.23098167283460497 .loss/FRt4/detection_loss‚2.968080125252406 1loss/SRqqt1/detection_loss‚1.6996914994190722 2loss/SRqqt1/assignment_loss‚1.5081049161506634 1loss/FBt2/assignment_loss‚0.18413300539403987 .loss/FBt1/detection_loss‚2.189019724726677 /loss/FBt2/detection_loss‚0.5765687726412374 /loss/FBt3/detection_loss‚2.5977400183677672 trainer/global_step‚9049 1loss/SRqqt4/assignment_loss‚2.061825688481331 1loss/SRqqt3/detection_loss‚1.9760782747733883 &loss/total_loss‚42.494646455504174 epoch‚7 _runtime‚7069.703627828 _step‚744 0loss/SRqqt4/detection_loss‚2.654067134857178 /loss/FRt1/detection_loss‚2.4864610062912105 0loss/FRt1/assignment_loss‚2.1548117250204086 2loss/SRqqt3/assignment_loss‚1.4414264412750148 0loss/FBt1/assignment_loss‚0.5265725896770164%,Â`Ý"Ú 1loss/FBt4/assignment_loss‚0.35034888481128623 2loss/SRqqt2/assignment_loss‚1.0133998342164194 1loss/FBt3/assignment_loss‚0.23098167283460497 .loss/FRt4/detection_loss‚2.968080125252406 1loss/SRqqt1/detection_loss‚1.6996914994190722 2loss/SRqqt1/assignment_loss‚1.5081049161506634 1loss/FBt2/assignment_loss‚0.18413300539403987 .loss/FBt1/detection_loss‚2.189019724726677 /loss/FBt2/detection_loss‚0.5765687726412374 /loss/FBt3/detection_loss‚2.5977400183677672 trainer/global_step‚9049 1loss/SRqqt4/assignment_loss‚2.061825688481331 1loss/SRqqt3/detection_loss‚1.9760782747733883 &loss/total_loss‚42.494646455504174 epoch‚7 _runtime‚7069.703627828 _step‚744 0loss/SRqqt4/detection_loss‚2.654067134857178 /loss/FRt1/detection_loss‚2.4864610062912105 0loss/FRt1/assignment_loss‚2.1548117250204086 2loss/SRqqt3/assignment_loss‚1.4414264412750148 0loss/FBt1/assignment_loss‚0.5265725896770164 .loss/FRt2/assignment_loss‚1.82472832247896 0loss/FRt3/assignment_loss‚3.5410339660114714 .loss/FRt3/detection_loss‚2.378612259361479 1loss/SRqqt2/detection_loss‚0.9705043812073878 .loss/FBt4/detection_loss‚2.346447695385326 0loss/FRt4/assignment_loss‚3.1160980979601542 /loss/FRt2/detection_loss‚1.6979104189096756 % _timestamp‚1.7884549781493049e+09è:¾š…“Þ"‚(:ˆ ÌÐæÔÆÀž× gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.364' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¹bEß"‚(:º ÌÐæÔÿÙÔÙ disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 194.94921875( proc.memory.availableMB‚ 951.01953125! disk.rbd2.out‚217.38330078125( disk.nvme0c0n1.out‚30610.47607421875 disk.drbd1777.out‚0$ memory_percent‚94.19543743133545 disk.dm-0.in‚ 194.94921875 disk.dm-1.out‚ 277.234375 disk.sda3.out‚ 277.234375 network.recv‚ 2.518804e+06# disk.dm-0.out‚30610.47607421875 disk.sda2.in‚0 disk.sda.out‚ 277.234375( proc.memory.percent‚17.5534725189209 cpu‚25.362084037693617) disk./.usagePercent‚74.51384604460577% disk./.usageGB‚1110.3101539611816 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.6796875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 194.94921875" proc.memory.rssMB‚ 2875.9609375& disk.nvme0n1.out‚30610.47607421875 network.sent‚ 2.1687948e+07 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd1777.in‚0¢+åžà"‚(:“ ÛÐæÔ´‡ÍÕ gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.942' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.188399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘DíY=á"‚(:² ÛÐæÔÚ„ò× disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875( disk.nvme0c0n1.out‚30708.63134765625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda1.in‚0 network.sent‚ 2.1795886e+07 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0& disk.nvme0n1.out‚30708.63134765625 disk.sda3.out‚278.0625) disk./.usagePercent‚74.51375516193112 disk.dm-1.out‚278.0625! disk.rbd2.out‚218.98095703125 disk.nvme0n1.in‚ 194.94921875 cpu‚25.361404594820566 network.recv‚ 2.524341e+06 disk.sda2.in‚0 disk.drbd1777.out‚0$ memory_percent‚94.19393539428711% disk./.usageGB‚1110.3087997436523 disk.rbd3.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 194.94921875 disk.dm-0.in‚ 194.94921875# disk.dm-0.out‚30708.63134765625& proc.memory.availableMB‚ 951.265625( proc.memory.percent‚17.5534725189209 disk.rbd1.out‚ 3.6796875 disk.sda.out‚278.0625 disk.sda1.out‚0ã$Üv2â"- _wandbruntime‚7104  _runtime‚7104qï‹Åã"¿ _runtime‚7104.784084314 _step‚745 % lr-AdamW/pg1‚3.22748345993059e-05 % lr-AdamW/pg2‚3.22748345993059e-05 trainer/global_step‚9099 % _timestamp‚1.7884550132296786e+09"¼AäÊä"Ä trainer/global_step‚9099 % _timestamp‚1.7884550132296786e+09 _runtime‚7104.784084314 _step‚745 % lr-AdamW/pg1‚3.22748345993059e-05 % lr-AdamW/pg2‚3.22748345993059e-05éå¶l2å"- _wandbruntime‚7104  _runtime‚7104»içç'æ"áO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884550132304797e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529071608703488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536691896235008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522800742076918 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536692570600448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171789657 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132318644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305534464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789657 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚515207819418624 trainer/global_step‚9099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132318642 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7104.785146346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570600448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7620961896960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305534464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928Ã^§7"ç"æO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304088261 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120680531 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚746 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚515207819418624 trainer/global_step‚9099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132318642 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7104.785146346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570600448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7620961896960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305534464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884550132304797e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529071608703488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536691896235008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522800742076918 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536692570600448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 17|Á±Àµ1789657 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132318644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305534464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789657 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120693244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚7105.454295135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577807217266688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788455013899223e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781338112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171808137 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585801823972352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585801623838720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120693719 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994606705664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642650 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚515262723549184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593067616142 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522855791165326 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120731 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054106 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068701696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7621124962304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43482312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7621124962304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414&‡6õ&ê"ïM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593067616142 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522855791165326 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120731 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054106 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068701696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7621124962304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43482312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7621124962304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171788073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120693244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _runtime‚7105.454295135 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577807217266688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788455013899223e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621781338112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171808137 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585801823972352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585801623838720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120693719 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994606705664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522856462642650 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚515262723549184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336ë>mJ2ë"- _wandbruntime‚7105  _runtime‚7105z‚íb_ì"Ù &loss/total_loss‚42.249293070664876 % _timestamp‚1.7884550139012277e+09 0loss/FRt1/assignment_loss‚2.6470029207895385 1loss/SRqqt4/assignment_loss‚2.216401797235012 /loss/FRt2/detection_loss‚1.9725749597308182 /loss/FBt3/detection_loss‚2.4920140419687544 0loss/FRt2/assignment_loss‚1.0458773303599584 1loss/SRqqt1/assignment_loss‚1.425210845803744 1loss/FBt2/assignment_loss‚0.13820252859595997 0loss/FRt3/assignment_loss‚3.3200718849897384 /loss/FRt4/detection_loss‚3.0396777341763177 epoch‚7 2loss/SRqqt3/assignment_loss‚1.6489526599273086 1loss/FBt3/assignment_loss‚0.07083141970049057 /loss/FBt1/detection_loss‚1.9337585330009461 /loss/FBt4/detection_loss‚3.9973475337028503 trainer/global_step‚9099 2loss/SRqqt2/assignment_loss‚1.3329167188953073 0loss/FBt4/assignment_loss‚0.0940667453687638 1loss/SRqqt4/detection_loss‚1.8844268536567688 /loss/FBt2/detection_loss‚0.7139564352979263 _runtime‚7105.455541204 _step‚748 .loss/FRt1/detection_loss‚2.136827457766488 1loss/SRqqt2/detection_loss‚1.0110309282599976 0loss/FRt4/assignment_loss‚3.3297382071614265 1loss/SRqqt3/detection_loss‚1.8300288939476013 0loss/FBt1/assignment_loss‚0.2152172550500829 /loss/FRt3/detection_loss‚2.2927640594542025 1loss/SRqqt1/detection_loss‚1.4603953258248799jŽdí"Þ /loss/FBt3/detection_loss‚2.4920140419687544 0loss/FRt2/assignment_loss‚1.0458773303599584 1loss/SRqqt1/assignment_loss‚1.425210845803744 1loss/FBt2/assignment_loss‚0.13820252859595997 0loss/FRt3/assignment_loss‚3.3200718849897384 /loss/FRt4/detection_loss‚3.0396777341763177 epoch‚7 2loss/SRqqt3/assignment_loss‚1.6489526599273086 1loss/FBt3/assignment_loss‚0.07083141970049057 /loss/FBt1/detection_loss‚1.9337585330009461 /loss/FBt4/detection_loss‚3.9973475337028503 trainer/global_step‚9099 2loss/SRqqt2/assignment_loss‚1.3329167188953073 0loss/FBt4/assignment_loss‚0.0940667453687638 1loss/SRqqt4/detection_loss‚1.8844268536567688 /loss/FBt2/detection_loss‚0.7139564352979263 _runtime‚7105.455541204 _step‚748 .loss/FRt1/detection_loss‚2.136827457766488 1loss/SRqqt2/detection_loss‚1.0110309282599976 0loss/FRt4/assignment_loss‚3.3297382071614265 1loss/SRqqt3/detection_loss‚1.8300288939476013 0loss/FBt1/assignment_loss‚0.2152172550500829 /loss/FRt3/detection_loss‚2.2927640594542025 1loss/SRqqt1/detection_loss‚1.4603953258248799 &loss/total_loss‚42.249293070664876 % _timestamp‚1.7884550139012277e+09 0loss/FRt1/assignment_loss‚2.6470029207895385 1loss/SRqqt4/assignment_loss‚2.216401797235012 /loss/FRt2/detection_loss‚1.9725749597308182ì—ñÁ’î"‚(:‡ êÐæÔýþÄÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oõ©Gï"‚(:¼ êÐæÔ³ëöØ disk.dm-1.out‚ 278.5078125 disk.drbd1777.out‚0 network.sent‚ 2.1802169e+07 disk.rbd3.in‚0( disk.nvme0c0n1.out‚30806.24365234375 disk.dm-1.in‚ 0.01171875% disk./.usageGB‚1110.3122062683105 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 278.5078125! disk.rbd2.out‚219.12158203125 disk.dm-0.in‚ 194.94921875 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2875.9609375 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 194.94921875 disk.nvme0n1.in‚ 194.94921875( proc.memory.percent‚17.5534725189209 proc.cpu.threads‚17) disk./.usagePercent‚74.51398377665919& disk.nvme0n1.out‚30806.24365234375 disk.drbd1777.in‚0 disk.sda2.out‚0$ memory_percent‚94.19550895690918 disk.rbd1.in‚0' proc.memory.availableMB‚ 951.0078125 cpu‚25.360777409142962 network.recv‚ 2.528747e+06 disk.sda.out‚ 278.5078125# disk.dm-0.out‚30806.243652343758v÷Ä“ð"‚(:ˆ ùÐæÔë™ÞÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.499' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 žÙFñ"‚(:» ùÐæÔÀï¤Ø disk.rbd1.out‚ 3.6796875& disk.nvme0n1.out‚30810.89208984375 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.360130139631885 network.recv‚ 2.534694e+06 disk.sda3.out‚ 279.3671875 disk.drbd1777.out‚0) disk./.usagePercent‚74.51398224061398$ disk./.usageGB‚1110.312183380127 disk.nvme0n1.in‚ 194.94921875 disk.sda2.out‚0' proc.memory.availableMB‚ 950.8671875( proc.memory.percent‚17.5534725189209 disk.sda1.in‚0$ memory_percent‚94.19636726379395 network.sent‚ 2.1910509e+07 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30810.89208984375 disk.dm-1.out‚ 279.3671875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 279.3671875 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 194.94921875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 194.94921875# disk.dm-0.out‚30810.89208984375" proc.memory.rssMB‚ 2875.9609375! disk.rbd2.out‚220.72314453125 disk.sda2.in‚0ËBá7“ò"‚(:ˆ ˆÑæÔþ¡ÆÕ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øæö¾Có"‚(:¸ ˆÑæÔóŸÆÙ& disk.nvme0n1.out‚30908.38427734375 disk.rbd1.out‚ 3.6796875$ memory_percent‚94.19715404510498# disk.dm-0.out‚30908.38427734375 cpu‚25.35947170683201 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30908.38427734375 disk.nvme0n1.in‚ 194.94921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 2.1916751e+07 network.recv‚ 2.539075e+06! disk.rbd2.out‚220.86376953125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.94921875 disk.rbd3.out‚0 disk.sda.out‚ 279.828125% disk./.usageGB‚1110.3121643066406 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 950.73828125" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 194.94921875 disk.sda3.out‚ 279.828125 disk.rbd2.in‚0 disk.dm-1.out‚ 279.828125 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda1.in‚0( disk./.usagePercent‚74.5139809605763‰øDD2ô"- _wandbruntime‚7140  _runtime‚7140*mãWÈõ" ' lr-AdamW/pg1‚3.1269951073342716e-05 ' lr-AdamW/pg2‚3.1269951073342716e-05 trainer/global_step‚9149 % _timestamp‚1.7884550492905867e+09 _runtime‚ 7140.84501806 _step‚7494ÃêÍö"Ç trainer/global_step‚9149 % _timestamp‚1.7884550492905867e+09 _runtime‚ 7140.84501806 _step‚749 ' lr-AdamW/pg1‚3.1269951073342716e-05 ' lr-AdamW/pg2‚3.1269951073342716e-05íýÝ2÷"- _wandbruntime‚7140  _runtime‚7140 ù“5@ ø"áO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525586112467210 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517953025946624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208348672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77611166 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8036892511744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7633740958342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580828103412224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036722596352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520586 _step‚750 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947702784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588864995923968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121315376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517953042985984 OEDeviceStatsMonitorjâq§.on_train_batch_start/allocation.small_pool.current‚20038 _runtime‚7140.846015249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133019442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7661291340288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7661291340288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43704210 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715113 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43703757 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7661947702784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133019444 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695671 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884550492914076e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121314921 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7633740958342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580828103412224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8036722596352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520586 _step‚750 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7661947702784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588864995923968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121315376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517953042985984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _runtime‚7140.846015249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133019442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7661291340288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7661291340288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43704210 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305715113 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43703757 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7661947702784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133019444 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172695671 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884550492914076e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121314921 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927324758016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚7141.521539996 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8037756061696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037594548224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43708198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121327988 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121327530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77619332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110768128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172714125 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531945613684736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531945613684736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393514 $ _timestamp‚1.788455049966757e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172714125 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9149 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588927510211584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539607706449920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521=v<(ü"ïM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121327988 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121327530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77619332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110768128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172714125 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531945613684736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172734189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531945613684736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393514 $ _timestamp‚1.788455049966757e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172714125 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9149 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588927510211584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539607706449920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305747583 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518007930077184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380828672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580889754149888 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927324758016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚7141.521539996 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8037756061696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8037594548224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43708198 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430ïT±›x2ý"- _wandbruntime‚7141  _runtime‚7141¿V)ç_þ"Ù 2loss/SRqqt1/assignment_loss‚1.1824193939055299 2loss/FBt1/assignment_loss‚0.002538708082283847 1loss/FBt4/assignment_loss‚0.17879528561818667 0loss/SRqqt4/detection_loss‚2.445833030868979 0loss/FRt3/assignment_loss‚2.8289525067806243 1loss/FBt2/assignment_loss‚0.23815293796553674 1loss/SRqqt1/detection_loss‚1.5360314996511328 0loss/FBt2/detection_loss‚0.47686349017273943 %loss/total_loss‚41.48985282848798 epoch‚7 2loss/SRqqt2/assignment_loss‚0.9497396138441914 .loss/FRt1/detection_loss‚2.309783323889687 % _timestamp‚1.7884550499687612e+09 /loss/FRt1/assignment_loss‚2.611638666618438 0loss/FRt4/assignment_loss‚3.4173751329553537 2loss/SRqqt3/assignment_loss‚1.4504105642437934 /loss/FBt1/detection_loss‚3.6153761595487595 trainer/global_step‚9149 1loss/FBt3/assignment_loss‚0.12229268194641918 /loss/FRt2/detection_loss‚1.8143433655019512 /loss/FBt4/detection_loss‚2.5929111459038476 _runtime‚7141.523058047 0loss/FRt2/assignment_loss‚0.9935807314786044 .loss/FRt3/detection_loss‚2.042517847418785 1loss/SRqqt4/assignment_loss‚1.896488425486228 /loss/FRt4/detection_loss‚2.9974702165044587 1loss/SRqqt2/detection_loss‚1.0068263083367663 1loss/SRqqt3/detection_loss‚1.7271064940094947 .loss/FBt3/detection_loss‚3.052405297756195 _step‚752>:¸°dÿ"Þ 0loss/FRt2/assignment_loss‚0.9935807314786044 .loss/FRt3/detection_loss‚2.042517847418785 1loss/SRqqt4/assignment_loss‚1.896488425486228 /loss/FRt4/detection_loss‚2.9974702165044587 1loss/SRqqt2/detection_loss‚1.0068263083367663 1loss/SRqqt3/detection_loss‚1.7271064940094947 .loss/FBt3/detection_loss‚3.052405297756195 _step‚752 2loss/SRqqt1/assignment_loss‚1.1824193939055299 2loss/FBt1/assignment_loss‚0.002538708082283847 1loss/FBt4/assignment_loss‚0.17879528561818667 0loss/SRqqt4/detection_loss‚2.445833030868979 0loss/FRt3/assignment_loss‚2.8289525067806243 1loss/FBt2/assignment_loss‚0.23815293796553674 1loss/SRqqt1/detection_loss‚1.5360314996511328 0loss/FBt2/detection_loss‚0.47686349017273943 %loss/total_loss‚41.48985282848798 epoch‚7 2loss/SRqqt2/assignment_loss‚0.9497396138441914 .loss/FRt1/detection_loss‚2.309783323889687 % _timestamp‚1.7884550499687612e+09 /loss/FRt1/assignment_loss‚2.611638666618438 0loss/FRt4/assignment_loss‚3.4173751329553537 2loss/SRqqt3/assignment_loss‚1.4504105642437934 /loss/FBt1/detection_loss‚3.6153761595487595 trainer/global_step‚9149 1loss/FBt3/assignment_loss‚0.12229268194641918 /loss/FRt2/detection_loss‚1.8143433655019512 /loss/FBt4/detection_loss‚2.5929111459038476 _runtime‚7141.523058047ð²oÏž€#‚(:“ —ÑæÔ¿¢ÆÖ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.374799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07D`Î@#‚(:µ —ÑæÔ°¨Ù disk.dm-1.in‚ 0.01171875% disk.nvme0n1.out‚30908.8701171875 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2875.9609375 disk.dm-1.out‚ 280.5234375 disk.sda1.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 194.94921875 cpu‚25.35886177864529 network.recv‚ 2.543181e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 280.5234375$ proc.memory.availableMB‚950.9375 proc.cpu.threads‚17" disk.dm-0.out‚30908.8701171875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0( proc.memory.percent‚17.5534725189209) disk./.usagePercent‚74.51398480068933% disk./.usageGB‚1110.3122215270996 disk.sda.out‚ 280.5234375 disk.sda.in‚ 0.01171875! disk.rbd2.out‚222.46142578125' disk.nvme0c0n1.out‚30908.8701171875 disk.drbd1777.out‚0$ memory_percent‚94.19593811035156 network.sent‚ 2.1922014e+07 disk.dm-0.in‚ 194.94921875 disk.nvme0n1.in‚ 194.949218752ž[Xž‚#‚(:“ ¦ÑæÔÅ«ÕÕ gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚152.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.429199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gŠ¿Fƒ#‚(:» ¦ÑæÔãõØ disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 194.94921875' disk.nvme0c0n1.out‚31006.3349609375 disk.drbd1777.in‚0 cpu‚25.358192657917332 proc.cpu.threads‚17 network.sent‚ 2.2030769e+07 disk.sda2.out‚0( disk./.usagePercent‚74.5104577848901" disk.dm-0.out‚31006.3349609375 disk.dm-1.out‚ 280.95703125" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚224.17626953125 disk.nvme0n1.in‚ 194.94921875$ disk./.usageGB‚1110.259666442871 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 280.95703125 disk.drbd1777.out‚0 disk.dm-0.in‚ 194.94921875 disk.rbd1.in‚0% disk.nvme0n1.out‚31006.3349609375 disk.dm-1.in‚ 0.01171875$ memory_percent‚94.19772624969482 disk.sda.out‚ 280.95703125 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 950.64453125 network.recv‚ 2.549487e+06 disk.rbd1.out‚ 3.6796875dä[92„#- _wandbruntime‚7176  _runtime‚7176¤Œ-È…# ' lr-AdamW/pg1‚3.0277753229912384e-05 ' lr-AdamW/pg2‚3.0277753229912384e-05 trainer/global_step‚9199 $ _timestamp‚1.788455085210656e+09 _runtime‚7176.765064918 _step‚753Hï™0͆#Ç ' lr-AdamW/pg1‚3.0277753229912384e-05 ' lr-AdamW/pg2‚3.0277753229912384e-05 trainer/global_step‚9199 $ _timestamp‚1.788455085210656e+09 _runtime‚7176.765064918 _step‚753ñ“Âs2‡#- _wandbruntime‚7176  _runtime‚7176ø*Ñå'ˆ#ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277146112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334618 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621723 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43932240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121951806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591990576881664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990380929024 trainer/global_step‚9199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43931770 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297750 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277146112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534706912588288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7673921859994 _runtime‚7176.766136087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079764573696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846096896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173641761 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846096896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583910640295424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚754 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933508608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811734 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133720242 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121951334 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414®¯cê'‰#äO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43931770 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674576297750 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277146112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534706912588288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7673921859994 _runtime‚7176.766136087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8079764573696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173641761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846096896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173641761 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542409846096896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583910640295424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚754 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7702933508608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811734 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133720242 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121951334 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277146112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334618 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520698232474624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621723 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43932240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520698249513984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133720244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121951806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591990576881664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591990380929024 trainer/global_step‚9199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088òBlî2Š#- _wandbruntime‚7177  _runtime‚7177D㑺‹#êM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542467018576896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294958 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173640177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43936292 @:DeviceStatsMonitor.on_train_ba¦9þ6Þ tch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7703752949760 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8080627004416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752949760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542467018576896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096573952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚7177.667956355 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096573952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520753136605184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121964024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788455086112811e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344198144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 _step‚755 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592052894097408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173640177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8080790615040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚534763247624192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053081648128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660241 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528427874900142 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121964493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520753153644544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173640177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583972267092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43936292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7703752949760 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8080627004416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7703752949760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542467018576896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7703096573952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚7177.667956355 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7703096573952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520753136605184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78027732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121964024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788455086112811e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583972291033088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344198144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 _step‚755 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592052894097408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173640177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8080790615040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚534763247624192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592053081648128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173660241 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528427874900142 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121964493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520753153644544 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116036361216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8123048843776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202026 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122588029 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122587564 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7743919314432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚758 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174567813 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116226022400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44160063 XJDeviceStatsMonitor.on_train_b¥r|·atch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 trainer/global_step‚9249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586993153238528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122883122688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚537524546527744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788455121498222e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ¯Îêë'™#åO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 trainer/global_step‚9249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586993153238528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8122883122688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚537524546527744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚537524546527744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788455121498222e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7715411637158 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545268483845120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174547775 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚537524564530688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134421042 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116036361216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8123048843776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202026 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122588029 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122587564 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7743919314432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚758 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174567813 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116226022400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44160063 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756ö&.ˆU2š#- _wandbruntime‚7213  _runtime‚7213…aáñ&›#ëM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122600680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122600214 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7716228072330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545325656325120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537580899566592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178732804096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531214588244874 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78436132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7744082379776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174566229 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44164546 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44164082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123904887808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884551221847985e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595178545253376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122600214 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7716228072330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545325656325120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537580899566592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595178732804096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531214588244874 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚537580917569536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78436132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7744082379776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174566229 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44164546 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44164082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8123904887808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884551221847985e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595178545253376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7784905120256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135121844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123223666 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚540342216473088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165874612736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123224111 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7784248757760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7784905120256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135121844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123223666 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚540342216473088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8165874612736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123224111 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728úÇ>ò2¬#- _wandbruntime‚7250  _runtime‚7250Ìqu¥'­#êM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123236824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175512345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44391837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724561408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175492281 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚763 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _runtime‚7250.200442199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166743428096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78844534 KDDeviceStatsMoóÉnitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚9299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123236369 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788455158645204e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628139 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304084287488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680á!õ&®#ïM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166743428096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78844534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚9299 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123236369 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788455158645204e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310628139 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304084287488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123236824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175512345 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44391837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724561408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175492281 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚763 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _runtime‚7250.200442199û³§¸2¯#- _wandbruntime‚7250  _runtime‚7250ª$X°#Ò 0loss/FRt4/assignment_loss‚3.4958062829642462 2loss/SRqqt4/assignment_loss‚1.5678862152355058 .loss/FRt2/detection_loss‚1.529344585686922 /loss/FBt1/detection_loss‚3.1539210230112076 -loss/FBt3/detection_loss‚2.89355460802714 0loss/FRt1/assignment_loss‚2.6856726538844224 .loss/FRt3/detection_loss‚2.036975439427034 % _timestamp‚1.7884551586472862e+09 .loss/FRt3/assignment_loss‚2.35710545530859 2loss/SRqqt1/assignment_loss‚1.3400772323920613 2loss/SRqqt2/assignment_loss‚0.9205561156676554 1loss/FBt4/assignment_loss‚0.19263981460244395 2loss/SRqqt3/assignment_loss‚1.1425821016013038 /loss/FBt2/detection_loss‚0.6523969951125695 .loss/FBt4/detection_loss‚2.529160092274348 %loss/total_loss‚42.16246456234519 trainer/global_step‚9299 _step‚764 0loss/FRt2/assignment_loss‚1.5834070646762848 1loss/FBt2/assignment_loss‚0.15983575869985447 /loss/FBt3/assignment_loss‚1.050656245749753 0loss/SRqqt1/detection_loss‚2.055534446700698 1loss/SRqqt2/detection_loss‚1.2094485041921708 1loss/SRqqt4/detection_loss‚2.4254847913980484 .loss/FRt1/detection_loss‚2.540220303506386 epoch‚7 _runtime‚7250.201607596 1loss/FBt1/assignment_loss‚0.19340551991263055 /loss/FRt4/detection_loss‚2.8835538769590445 1loss/SRqqt3/detection_loss‚1.5632394353548686¯i‹]±#× .loss/FRt1/detection_loss‚2.540220303506386 epoch‚7 _runtime‚7250.201607596 1loss/FBt1/assignment_loss‚0.19340551991263055 /loss/FRt4/detection_loss‚2.8835538769590445 1loss/SRqqt3/detection_loss‚1.5632394353548686 0loss/FRt4/assignment_loss‚3.4958062829642462 2loss/SRqqt4/assignment_loss‚1.5678862152355058 .loss/FRt2/detection_loss‚1.529344585686922 /loss/FBt1/detection_loss‚3.1539210230112076 -loss/FBt3/detection_loss‚2.89355460802714 0loss/FRt1/assignment_loss‚2.6856726538844224 .loss/FRt3/detection_loss‚2.036975439427034 % _timestamp‚1.7884551586472862e+09 .loss/FRt3/assignment_loss‚2.35710545530859 2loss/SRqqt1/assignment_loss‚1.3400772323920613 2loss/SRqqt2/assignment_loss‚0.9205561156676554 1loss/FBt4/assignment_loss‚0.19263981460244395 2loss/SRqqt3/assignment_loss‚1.1425821016013038 /loss/FBt2/detection_loss‚0.6523969951125695 .loss/FBt4/detection_loss‚2.529160092274348 %loss/total_loss‚42.16246456234519 trainer/global_step‚9299 _step‚764 0loss/FRt2/assignment_loss‚1.5834070646762848 1loss/FBt2/assignment_loss‚0.15983575869985447 /loss/FBt3/assignment_loss‚1.050656245749753 0loss/SRqqt1/detection_loss‚2.055534446700698 1loss/SRqqt2/detection_loss‚1.2094485041921708 1loss/SRqqt4/detection_loss‚2.4254847913980484ütùdÌ’²#‚(:‡ €ÒæÔ¡ÐÍÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07T£GD³#‚(:¹ €ÒæÔ¡íÖÙ cpu‚25.354339221635307 proc.cpu.threads‚17& disk.nvme0n1.out‚31398.59912109375 network.recv‚ 2.593213e+06 disk.rbd3.out‚0 disk.sda.out‚ 284.6328125 disk.sda1.out‚0 disk.sda3.out‚ 284.6328125 disk.rbd0.out‚0% proc.memory.availableMB‚ 950.15625" disk.nvme0c0n1.in‚ 194.96484375 disk.sda2.in‚0 network.sent‚ 2.2277952e+07 disk.dm-1.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1110.2633743286133 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875# memory_percent‚94.2007064819336( proc.memory.percent‚17.5534725189209! disk.rbd2.out‚229.33642578125( disk.nvme0c0n1.out‚31398.59912109375 disk.dm-1.out‚ 284.6328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2875.9609375) disk./.usagePercent‚74.51070662421338 disk.dm-0.in‚ 194.96484375# disk.dm-0.out‚31398.59912109375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 194.96484375tšt´#‚(:… ÒæÔ§¢æÕ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚95.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-&:WCµ#‚(:¸ ÒæÔÑú•Ø) disk./.usagePercent‚74.51070918428873 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 194.96484375 disk.rbd1.out‚ 3.6796875 disk.rbd3.in‚0( disk.nvme0c0n1.out‚31399.14794921875# disk.dm-0.out‚31399.14794921875 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 194.96484375 disk.nvme0n1.in‚ 194.96484375( proc.memory.percent‚17.5534725189209$ disk./.usageGB‚1110.263412475586 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0$ memory_percent‚94.20080184936523& proc.memory.availableMB‚ 950.140625 disk.rbd1.in‚0 disk.sda1.out‚0 network.recv‚ 2.599719e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚31399.14794921875" proc.memory.rssMB‚ 2875.9609375 cpu‚25.35369055788466 network.sent‚ 2.238579e+07 disk.sda.in‚ 0.01171875 disk.sda.out‚ 285.4140625 disk.sda3.out‚ 285.4140625! disk.rbd2.out‚230.91064453125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 285.4140625 disk.rbd2.in‚0MB8H2¶#- _wandbruntime‚7285  _runtime‚7285Q ¦Ç·#Á _runtime‚7285.723316868 _step‚765 & lr-AdamW/pg1‚2.737932804388038e-05 & lr-AdamW/pg2‚2.737932804388038e-05 trainer/global_step‚9349 % _timestamp‚1.7884551941688998e+09g€{ñ̸#Æ & lr-AdamW/pg2‚2.737932804388038e-05 trainer/global_step‚9349 % _timestamp‚1.7884551941688998e+09 _runtime‚7285.723316868 _step‚765 & lr-AdamW/pg1‚2.737932804388038e-05ý¨îµó2¹#- _wandbruntime‚7285  _runtime‚7285šG*ç'º#áO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528933869097984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7825234563584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123859745 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8208935765504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601367162770432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536730951413958 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚7285.724281967 _step‚766 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528933852058624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7825234563584 % _timestamp‚1.7884551941697028e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176399879 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123860194 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543159850412544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176419917 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79244764 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚9349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7825890926080 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593158227004928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7796427878218 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7825234563584 % _timestamp‚1.7884551941697028e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176399879 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatu„ô$sMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123860194 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543159850412544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176419917 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79244764 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚9349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7825890926080 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593158227004928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7796427878218 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884551948558936e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931821568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312254991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚536786671979690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9349 _runtime‚7286.410953031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418333 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135836660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176438397 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528988773228544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543216203451392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543216203451392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚767 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123872432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209963412480 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601429865095168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429683835904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543216221454336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44619500 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182÷U €ö&¾#ðM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176418333 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135836660 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176438397 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528988773228544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543216221454336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543216203451392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543216203451392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚767 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123872432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8209963412480 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601429865095168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601429683835904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543216221454336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44619500 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176418333 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚536786000502366 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79252932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751146 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884551948558936e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931821568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312254991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚536786671979690 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9349 _runtime‚7286.410953031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064ÿ0'P2¿#- _wandbruntime‚7286  _runtime‚7286&yj ZÀ#Ô 2loss/SRqqt1/assignment_loss‚1.3041951632872224 2loss/SRqqt4/assignment_loss‚2.0239169895648956 0loss/FBt2/assignment_loss‚0.1286116238598997 /loss/FRt3/detection_loss‚2.3182784077667056 .loss/FBt4/detection_loss‚2.229177989065647 0loss/FRt1/assignment_loss‚2.4042152823234093 epoch‚7 /loss/FRt4/detection_loss‚2.8106238226095837 .loss/FBt1/detection_loss‚3.023342955112457 trainer/global_step‚9349 /loss/FRt2/assignment_loss‚1.586650162935257 0loss/FRt3/assignment_loss‚2.6743655857585726 /loss/FRt2/detection_loss‚1.8720910819652288 1loss/SRqqt1/detection_loss‚1.8421778689324855 .loss/FRt1/detection_loss‚2.172451339662075 0loss/FRt4/assignment_loss‚3.9980721195538838 2loss/SRqqt2/assignment_loss‚0.8527058826992289 1loss/FBt1/assignment_loss‚0.42004588670679227 0loss/FBt4/assignment_loss‚0.3617529545153957 0loss/SRqqt3/detection_loss‚1.626773846273621 1loss/SRqqt4/detection_loss‚2.5673131385157184 %loss/total_loss‚41.67565675671916 % _timestamp‚1.7884551948578522e+09 1loss/SRqqt3/assignment_loss‚1.388695331616327 _runtime‚7286.412153874 _step‚768 0loss/FBt3/assignment_loss‚0.4185823594452813 1loss/SRqqt2/detection_loss‚0.9312375362962484 /loss/FBt2/detection_loss‚0.8563953277326095 /loss/FBt3/detection_loss‚1.8639841005206108-Èñâ_Á#Ù .loss/FRt1/detection_loss‚2.172451339662075 0loss/FRt4/assignment_loss‚3.9980721195538838 2loss/SRqqt2/assignment_loss‚0.8527058826992289 1loss/FBt1/assignment_loss‚0.42004588670679227 0loss/FBt4/assignment_loss‚0.3617529545153957 0loss/SRqqt3/detection_loss‚1.626773846273621 1loss/SRqqt4/detection_loss‚2.5673131385157184 %loss/total_loss‚41.67565675671916 % _timestamp‚1.7884551948578522e+09 1loss/SRqqt3/assignment_loss‚1.388695331616327 _runtime‚7286.412153874 _step‚768 0loss/FBt3/assignment_loss‚0.4185823594452813 1loss/SRqqt2/detection_loss‚0.9312375362962484 /loss/FBt2/detection_loss‚0.8563953277326095 /loss/FBt3/detection_loss‚1.8639841005206108 2loss/SRqqt1/assignment_loss‚1.3041951632872224 2loss/SRqqt4/assignment_loss‚2.0239169895648956 0loss/FBt2/assignment_loss‚0.1286116238598997 /loss/FRt3/detection_loss‚2.3182784077667056 .loss/FBt4/detection_loss‚2.229177989065647 0loss/FRt1/assignment_loss‚2.4042152823234093 epoch‚7 /loss/FRt4/detection_loss‚2.8106238226095837 .loss/FBt1/detection_loss‚3.023342955112457 trainer/global_step‚9349 /loss/FRt2/assignment_loss‚1.586650162935257 0loss/FRt3/assignment_loss‚2.6743655857585726 /loss/FRt2/detection_loss‚1.8720910819652288 1loss/SRqqt1/detection_loss‚1.8421778689324855€ZOÄSžÂ#‚(:“ žÒæÔòàêÕ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.646' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.529199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%høïžÃ#‚(:º žÒæÔô½¤Ø disk.sda.out‚ 285.92578125 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚94.1995620727539( proc.memory.percent‚£X€N§17.5534725189209 disk.dm-0.in‚ 194.96484375" disk.nvme0c0n1.in‚ 194.96484375% proc.memory.availableMB‚ 950.34375 network.sent‚ 2.2392132e+07 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 285.92578125! disk.rbd2.out‚231.01220703125 disk.dm-1.out‚ 285.92578125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚31496.64404296875" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 network.recv‚ 2.604168e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875( disk./.usagePercent‚74.5107053441757$ disk./.usageGB‚1110.263355255127 disk.drbd1777.in‚0# disk.dm-0.out‚31496.64404296875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 cpu‚25.353032842410965& disk.nvme0n1.out‚31496.64404296875 disk.nvme0n1.in‚ 194.96484375 disk.sda2.in‚0 disk.sda1.in‚0]$oÛ“Ä#‚(:ˆ ­ÒæÔßöçÕ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊvÿAÅ#‚(:¶ ­ÒæÔ‹´ÓØ disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 286.5703125$ proc.memory.availableMB‚950.3125 disk.sda.out‚ 286.5703125& disk.nvme0n1.out‚31594.57666015625 disk.sda2.out‚0 cpu‚25.35244426337797 disk.sda3.in‚ 0.01171875 network.recv‚ 2.611733e+06 disk.sda1.out‚0! disk.rbd2.out‚232.58642578125$ memory_percent‚94.19975280761719 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 disk.sda3.out‚ 286.5703125( disk./.usagePercent‚74.5107053441757 disk.rbd1.out‚ 3.6796875 disk.rbd3.in‚0 disk.dm-0.in‚ 194.99609375# disk.dm-0.out‚31594.57666015625 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 194.99609375 network.sent‚ 2.2500261e+07 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 194.99609375( disk.nvme0c0n1.out‚31594.57666015625( proc.memory.percent‚17.5534725189209$ disk./.usageGB‚1110.263355255127Ëéü‡“Æ#‚(:ˆ ¼ÒæÔ‡¢ÎÕ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.939' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÁÇDÇ#‚(:¹ ¼ÒæÔ‡ý°Ø disk.drbd1777.out‚0# disk.dm-0.out‚31594.82275390625 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 287.40234375 disk.sda1.in‚0( proc.memory.percent‚17.5534725189209 disk.drbd1777.in‚0 disk.dm-0.in‚ 194.99609375 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 cpu‚25.35182862785487 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚74.51070688022091 disk.nvme0n1.in‚ 194.99609375 disk.sda3.out‚ 287.40234375$ proc.memory.availableMB‚950.3125" proc.memory.rssMB‚ 2875.9609375! disk.rbd2.out‚232.75439453125 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚94.19975280761719 disk.dm-1.out‚ 287.40234375 network.recv‚ 2.616707e+06% disk./.usageGB‚1110.2633781433105 disk.rbd0.in‚0( disk.nvme0c0n1.out‚31594.82275390625& disk.nvme0n1.out‚31594.82275390625 disk.sda2.in‚0 disk.sda3.in‚ 0.01171875 network.sent‚ 2.25068e+07" disk.nvme0c0n1.in‚ 194.99609375 #S2È#- _wandbruntime‚7322  _runtime‚7322føÉÉ#à % _timestamp‚1.7884552305836792e+09 _runtime‚7322.138056177 _step‚769 ' lr-AdamW/pg1‚2.6439913576245723e-05 ' lr-AdamW/pg2‚2.6439913576245723e-05 trainer/global_step‚9399ù®äÎÊ#È _step‚769 ' lr-AdamW/pg1‚2.6439913576245723e-05 ' lr-AdamW/pg2‚2.6439913576245723e-05 trainer/global_step‚9399 % _timestamp‚1.7884552305836792e+09 _runtime‚7322.138056177xrº2Ë#- _wandbruntime‚7322  _runtime‚7322iä‚7ç'Ì#áO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545977502354944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325931 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44843500 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7866220369408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7322.139451787 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876731904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124496666 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 í9ì'Í#æO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136523442 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545977520357888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44843068 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7866220369408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124496232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8252142682624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251987447296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545977502354944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124496666 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545977502354944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136523442 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325931 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44843500 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7866220369408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7322.139451787 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876731904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚9399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553900895191040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539572042369774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039797248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 _step‚771 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7322.811739703 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253021299200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7838079652590 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252868174336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364449 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344385 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44848002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555459865088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7867039797248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7867696173056 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124508894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7966133Ä•Çîž2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136537458 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884552312568223e+09á,¤+ö&Ð#ðM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7867039797248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7867696173056 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124508894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847098 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136537458 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884552312568223e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553900895191040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539572042369774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039797248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 _step‚771 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚7322.811739703 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253021299200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7838079652590 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252868174336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177364449 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344385 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44848002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555459865088ƒ|›#å2Ñ#- _wandbruntime‚7322  _runtime‚7322ÞZaÈ\Ò#Ö .loss/FRt3/detection_loss‚2.318131837040879 /loss/FBt2/detection_loss‚0.5334886795668027 _runtime‚7322.812793429 /loss/FRt1/assignment_loss‚2.388473904651144 0loss/FRt2/assignment_loss‚1.3244056296798419 2loss/SRqqt4/assignment_loss‚1.6066993087016304 .loss/FRt4/detection_loss‚2.864329405733057 1loss/SRqqt1/detection_loss‚1.6212455268230355 epoch‚7 1loss/FBt1/assignment_loss‚0.14722880899353186 1loss/FBt2/assignment_loss‚0.10758235975511364 .loss/FRt2/detection_loss‚1.558982193610578 .loss/FRt1/detection_loss‚2.344207804488099 1loss/SRqqt2/detection_loss‚0.9318976211087668 /loss/FBt3/detection_loss‚2.7911963860193887 $ _timestamp‚1.788455231258465e+09 0loss/FRt4/assignment_loss‚3.6881452379999935 2loss/SRqqt1/assignment_loss‚1.4434910907263034 2loss/SRqqt2/assignment_loss‚0.8301871744368006 1loss/FBt3/assignment_loss‚0.10189621266908826 1loss/SRqqt3/detection_loss‚1.5063484767079354 %loss/total_loss‚41.30920909925024 trainer/global_step‚9399 0loss/FRt3/assignment_loss‚3.5414534557697386 1loss/SRqqt4/detection_loss‚2.5972475070377876 /loss/FBt1/detection_loss‚2.1874857544898987 _step‚772 0loss/FBt4/assignment_loss‚0.1748960881959647 /loss/FBt4/detection_loss‚3.0199407849993025 2loss/SRqqt3/assignment_loss‚1.6802478500455618(øèÊaÓ#Û 1loss/SRqqt3/detection_loss‚1.5063484767079354 %loss/total_loss‚41.30920909925024 trainer/global_step‚9399 0loss/FRt3/assignment_loss‚3.5414534557697386 1loss/SRqqt4/detection_loss‚2.5972475070377876 /loss/FBt1/detection_loss‚2.1874857544898987 _step‚772 0loss/FBt4/assignment_loss‚0.1748960881959647 /loss/FBt4/detection_loss‚3.0199407849993025 2loss/SRqqt3/assignment_loss‚1.6802478500455618 .loss/FRt3/detection_loss‚2.318131837040879 /loss/FBt2/detection_loss‚0.5334886795668027 _runtime‚7322.812793429 /loss/FRt1/assignment_loss‚2.388473904651144 0loss/FRt2/assignment_loss‚1.3244056296798419 2loss/SRqqt4/assignment_loss‚1.6066993087016304 .loss/FRt4/detection_loss‚2.864329405733057 1loss/SRqqt1/detection_loss‚1.6212455268230355 epoch‚7 1loss/FBt1/assignment_loss‚0.14722880899353186 1loss/FBt2/assignment_loss‚0.10758235975511364 .loss/FRt2/detection_loss‚1.558982193610578 .loss/FRt1/detection_loss‚2.344207804488099 1loss/SRqqt2/detection_loss‚0.9318976211087668 /loss/FBt3/detection_loss‚2.7911963860193887 $ _timestamp‚1.788455231258465e+09 0loss/FRt4/assignment_loss‚3.6881452379999935 2loss/SRqqt1/assignment_loss‚1.4434910907263034 2loss/SRqqt2/assignment_loss‚0.8301871744368006 1loss/FBt3/assignment_loss‚0.10189621266908826„jÏžÔ#‚(:“ ËÒæÔôÏÞÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¼ èJÕ#‚(:¿ ËÒæÔÀç“Ù disk.rbd3.out‚0) disk./.usagePercent‚74.51070560018324 disk.sda3.out‚ 287.95703125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0# disk.dm-0.out‚31692.47119140625 disk.dm-1.out‚ 287.95703125( proc.memory.availableMB‚ 950.05859375( proc.memory.percent‚17.5534725189209 proc.cpu.threads‚17% disk./.usageGB‚1110.2633590698242! disk.rbd2.out‚234.32861328125& disk.nvme0n1.out‚31692.47119140625 cpu‚25.35117401126106 disk.rbd2.in‚0 disk.nvme0n1.in‚ 194.99609375 disk.sda2.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 194.99609375 network.recv‚ 2.620738e+06 disk.rbd0.out‚0 disk.sda.in‚ 0.01171875 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0" proc.memory.rssMB‚ 2875.9609375 network.sent‚ 2.2512058e+07 disk.dm-0.in‚ 194.99609375$ memory_percent‚94.20130252838135 disk.sda1.in‚0( disk.nvme0c0n1.out‚31692.47119140625 disk.drbd1777.out‚0 disk.sda.out‚ 287.95703125wTu…žÖ#‚(:“ ÚÒæÔó‰ÛÕ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.685' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.737000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O ÖŽE×#‚(:º ÚÒæÔÕª¦Ù% disk./.usageGB‚1110.2633590698242 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 288.390625( proc.memory.availableMB‚ 950.03515625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 288.390625 disk.rbd3.out‚0 network.sent‚ 2.2620962e+07 disk.drbd1777.in‚0 disk.sda.out‚ 288.390625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 194.99609375( disk.nvme0c0n1.out‚31692.78662109375# disk.dm-0.out‚31692.78662109375 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875( proc.memory.percent‚17.5534725189209& disk.nvme0n1.out‚31692.78662109375 disk.sda2.out‚0 disk.rbd1.out‚ 3.6796875 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0$ memory_percent‚94.20144557952881 cpu‚25.350594424133075! disk.rbd2.out‚234.49267578125 network.recv‚ 2.628047e+06) disk./.usagePercent‚74.51070560018324 disk.nvme0n1.in‚ 194.99609375 disk.dm-0.in‚ 194.99609375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2875.9609375 disk.rbd3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17΀úZ2Ø#- _wandbruntime‚7358  _runtime‚7358ÿ¯­æÉÙ#à ' lr-AdamW/pg1‚2.5514191363718925e-05 ' lr-AdamW/pg2‚2.5514191363718925e-05 trainer/global_step‚9449 % _timestamp‚1.7884552666815906e+09 _runtime‚7358.235973026 _step‚773Ô‡DÎÚ#È _step‚773 ' lr-AdamW/pg1‚2.5514191363718925e-05 ' lr-AdamW/pg2‚2.5514191363718925e-05 trainer/global_step‚9449 % _timestamp‚1.7884552666815906e+09 _runtime‚7358.235973026…º÷c2Û#- _wandbruntime‚7358  _runtime‚7358z W*ç'Ü#áO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534424282153984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚556703034838016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚7358.237068851 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9449 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125133601 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137224244 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401dqgÝ#æO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994790 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚556703034838016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _runtime‚7358.237068851 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9449 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125133601 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.aüæ§$…ll.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137224244 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534424282153984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844†&nÈÏ2Þ#- _wandbruntime‚7358  _runtime‚7358¸ÕAñ&ß#ëM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025603072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45076591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚775 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508695 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908681978880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680997287424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80069734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207318016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7908025603072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290501 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607681176449536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45077032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296201000448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125146323 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681978880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8296045778432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9449Ì»Á›ö&à#ðM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45077032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296201000448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125146323 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681978880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8296045778432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7908025603072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45076591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚775 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508695 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908681978880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607680997287424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80069734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207318016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7908025603072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290501 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607681176449536‡ ô2á#- _wandbruntime‚7358  _runtime‚7358£ðÕ_\â#Ö /loss/FRt1/detection_loss‚2.3676606921049266 /loss/FBt1/detection_loss‚2.4898268580436707 /loss/FBt4/detection_loss‚2.6331678087061103 trainer/global_step‚9449 2loss/SRqqt3/assignment_loss‚1.3894354697716023 0loss/FBt1/assignment_loss‚0.4251855284201156 /loss/FBt4/assignment_loss‚0.431090428948747 /loss/FRt2/detection_loss‚1.8479492273647338 1loss/SRqqt1/detection_loss‚1.4677787501132116 /loss/FBt2/detection_loss‚0.8607414076803672 _runtime‚7358.923202499 0loss/FRt1/assignment_loss‚2.2418785859376955 0loss/FRt2/assignment_loss‚1.2682364627718925 .loss/FRt4/detection_loss‚2.971783420135235 /loss/FBt3/detection_loss‚2.2369522100145165 %loss/total_loss‚41.79843143295908 2loss/SRqqt2/assignment_loss‚0.9425781357636237 1loss/SRqqt2/detection_loss‚0.9894845918478334 1loss/SRqqt3/detection_loss‚1.8215433085958164 1loss/SRqqt4/detection_loss‚2.6517183573349663 0loss/FRt4/assignment_loss‚3.7311296216372787 .loss/FRt3/detection_loss‚2.171466601099454 1loss/FBt2/assignment_loss‚0.20636204753216908 epoch‚7 % _timestamp‚1.7884552673689144e+09 _step‚776 2loss/SRqqt1/assignment_loss‚1.1464867445465643 2loss/SRqqt4/assignment_loss‚2.1703872389119603 0loss/FBt3/assignment_loss‚0.4331051210280169 /loss/FRt3/assignment_loss‚2.902482814648572ØË,aã#Û 2loss/SRqqt1/assignment_loss‚1.1464867445465643 2loss/SRqqt4/assignment_loss‚2.1703872389119603 0loss/FBt3/assignment_loss‚0.4331051210280169 /loss/FRt3/assignment_loss‚2.902482814648572 /loss/FRt1/detection_loss‚2.3676606921049266 /loss/FBt1/detection_loss‚2.4898268580436707 /loss/FBt4/detection_loss‚2.6331678087061103 trainer/global_step‚9449 2loss/SRqqt3/assignment_loss‚1.3894354697716023 0loss/FBt1/assignment_loss‚0.4251855284201156 /loss/FBt4/assignment_loss‚0.431090428948747 /loss/FRt2/detection_loss‚1.8479492273647338 1loss/SRqqt1/detection_loss‚1.4677787501132116 /loss/FBt2/detection_loss‚0.8607414076803672 _runtime‚7358.923202499 0loss/FRt1/assignment_loss‚2.2418785859376955 0loss/FRt2/assignment_loss‚1.2682364627718925 .loss/FRt4/detection_loss‚2.971783420135235 /loss/FBt3/detection_loss‚2.2369522100145165 %loss/total_loss‚41.79843143295908 2loss/SRqqt2/assignment_loss‚0.9425781357636237 1loss/SRqqt2/detection_loss‚0.9894845918478334 1loss/SRqqt3/detection_loss‚1.8215433085958164 1loss/SRqqt4/detection_loss‚2.6517183573349663 0loss/FRt4/assignment_loss‚3.7311296216372787 .loss/FRt3/detection_loss‚2.171466601099454 1loss/FBt2/assignment_loss‚0.20636204753216908 epoch‚7 % _timestamp‚1.7884552673689144e+09 _step‚776ˆíŒbžä#‚(:“ éÒæÔï–ÍÕ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.423000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ÍDå#‚(:¹ éÒæÔËÍÙ( proc.memory.percent‚17.5534725189209) disk./.usagePercent‚74.51070560018324 disk.sda1.out‚0 disk.sda3.out‚ 289.140625 disk.rbd0.out‚0$ memory_percent‚94.20015811920166 network.sent‚ 2.2626902e+07 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 289.140625( disk.nvme0c0n1.out‚31790.65087890625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0! disk.rbd2.out‚236.06689453125 cpu‚25.34998218070383# disk.dm-0.out‚31790.65087890625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.sda2.out‚0( proc.memory.availableMB‚ 950.24609375 disk.dm-1.out‚ 289.140625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 194.99609375" proc.memory.rssMB‚ 2875.9609375 proc.cpu.threads‚17 network.recv‚ 2.632409e+06" disk.nvme0c0n1.in‚ 194.99609375 disk.dm-0.in‚ 194.99609375 disk.sda2.in‚0% disk./.usageGB‚1110.2633590698242 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚31790.65087890625?t—“æ#‚(:ˆ øÒæÔà¶÷Õ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä }»Dç#‚(:¹ øÒæÔ²×ÁØ& disk.nvme0n1.out‚31888.16748046875 disk.drbd1777.out‚0 disk.rbd3.out‚0( proc.memory.percent‚17.5534725189209 network.sent‚ 2.2739736e+07 disk.rbd2.in‚0 disk.dm-0.in‚ 194.99609375 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0! disk.rbd2.out‚237.74658203125$ memory_percent‚94.19865608215332' proc.memory.availableMB‚ 950.4921875" proc.memory.rssMB‚ 2875.9609375 network.recv‚ 2.646964e+06% disk./.usageGB‚1110.2633781433105 disk.rbd1.out‚ 3.6796875 disk.rbd3.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚31888.16748046875) disk./.usagePercent‚74.51070688022091 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 289.796875" disk.nvme0c0n1.in‚ 194.99609375( disk.nvme0c0n1.out‚31888.16748046875 cpu‚25.349348514126746 disk.dm-1.out‚ 289.796875 disk.dm-1.in‚ 0.01171875 disk.sda.out‚ 289.796875 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.nvme0n1.in‚ 194.99609375 Ñui2è#- _wandbruntime‚7394  _runtime‚7394Ë´Ë+Èé# ' lr-AdamW/pg1‚2.4602354325421575e-05 ' lr-AdamW/pg2‚2.4602354325421575e-05 trainer/global_step‚9499 $ _timestamp‚1.788455303309908e+09 _runtime‚7394.864351656 _step‚777íù¾Íê#Ç ' lr-AdamW/pg1‚2.4602354325421575e-05 ' lr-AdamW/pg2‚2.4602354325421575e-05 trainer/global_step‚9499 $ _timestamp‚1.788455303309908e+09 _runtime‚7394.864351656 _step‚777‰}¦ì#2ë#- _wandbruntime‚7394  _runtime‚7394Ú® PQì#áO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceÞmnÕ–'StatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8338421777408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559561672586240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602405861594624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884553033106992e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125771696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198073 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125771257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚7394.865386316 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137925044 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338270736384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744283372032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016182 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539066 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744108390912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45301293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0³® ì'í#æO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125771696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179198073 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125771257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚7394.865386316 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137925044 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338270736384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137925044 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744283372032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016182 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545088405539066 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744108390912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45301293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8338421777408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559561672586240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602405861594624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884553033106992e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537169471642624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439Š9{Гî#‚(:ˆ ‡ÓæÔ¸èÝÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.299' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o¸¦Hï#‚(:½ ‡ÓæÔý€Ù disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 195.02734375 disk.drbd1777.in‚0$ memory_percent‚94.20008659362793' proc.memory.availableMB‚ 950.2578125" proc.memory.rssMB‚ 2875.9609375( proc.memory.percent‚17.5534725189209 network.sent‚ 2.2744938e+07 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 195.02734375 proc.cpu.threads‚17 disk.rbd1.out‚ 3.6796875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚31891.24267578125 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0! disk.rbd2.out‚237.74658203125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.out‚ 291.13671875 disk.dm-0.in‚ 195.02734375 network.recv‚ 2.650602e+06$ disk./.usageGB‚1110.261100769043 disk.rbd3.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 291.13671875) disk./.usagePercent‚74.51055404372298 disk.sda.out‚ 291.13671875& disk.nvme0n1.out‚31891.24267578125# disk.dm-0.out‚31891.24267578125 cpu‚25.34877283461278ºkMB2ð#- _wandbruntime‚7395  _runtime‚7395cV.·ñ&ñ#ëM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125784384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80478134 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339289582080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011408896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806610169344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317135547 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7949011408896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551669159278592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125783938 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305806 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610806801914368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339121777152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚779 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667784704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545144126104590 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884553039898455e+09 trainer/global_step‚9499 _runtime‚7395.544755883 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581914 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752R‚ÚÓOò#ðM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚779 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7949667784704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45306250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.ô›Rà§#on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545144126104590 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884553039898455e+09 trainer/global_step‚9499 _runtime‚7395.544755883 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581914 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125784384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80478134 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339289582080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7949011408896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610806610169344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317135547 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7949011408896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551669159278592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551669177281536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125783938 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45305806 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610806801914368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339121777152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944‹ÎÔ2ó#- _wandbruntime‚7395  _runtime‚7395u\ô#Ö 0loss/FRt2/assignment_loss‚1.0558694154024124 /loss/FRt3/detection_loss‚2.0624601602250214 1loss/SRqqt3/detection_loss‚1.7623912502594035 /loss/FBt3/detection_loss‚2.9346781075000763 %loss/total_loss‚40.84616115279447 2loss/SRqqt4/assignment_loss‚2.3028030078858137 0loss/FBt4/assignment_loss‚0.4917383852570007 /loss/FRt1/detection_loss‚2.2813474969430403 0loss/SRqqt4/detection_loss‚2.542478329191605 /loss/FBt1/detection_loss‚2.4807360768318176 .loss/FBt4/detection_loss‚2.057953159014384 epoch‚7 0loss/FRt1/assignment_loss‚2.5248793946071104 0loss/FRt4/assignment_loss‚3.3062512786300093 2loss/SRqqt1/assignment_loss‚1.2262608967139386 _runtime‚7395.545850005 /loss/FRt2/detection_loss‚1.6074681142677687 .loss/FRt4/detection_loss‚3.118605845504337 % _timestamp‚1.7884553039914863e+09 1loss/FBt2/assignment_loss‚0.15778601939141687 1loss/SRqqt1/detection_loss‚1.5387958324115192 trainer/global_step‚9499 /loss/FRt3/assignment_loss‚2.277902221193119 2loss/SRqqt3/assignment_loss‚1.1465032226566612 0loss/FBt3/assignment_loss‚0.3138676499656867 1loss/SRqqt2/detection_loss‚1.0269704540260136 1loss/SRqqt2/assignment_loss‚1.129094834002899 0loss/FBt2/detection_loss‚0.48878270803236623 _step‚780 0loss/FBt1/assignment_loss‚1.0105372928810539 Õaõ#Û 1loss/FBt2/assignment_loss‚0.15778601939141687 1loss/SRqqt1/detection_loss‚1.5387958324115192 trainer/global_step‚9499 /loss/FRt3/assignment_loss‚2.277902221193119 2loss/SRqqt3/assignment_loss‚1.1465032226566612 0loss/FBt3/assignment_loss‚0.3138676499656867 1loss/SRqqt2/detection_loss‚1.0269704540260136 1loss/SRqqt2/assignment_loss‚1.129094834002899 0loss/FBt2/detection_loss‚0.48878270803236623 _step‚780 0loss/FBt1/assignment_loss‚1.0105372928810539 0loss/FRt2/assignment_loss‚1.0558694154024124 /loss/FRt3/detection_loss‚2.0624601602250214 1loss/SRqqt3/detection_loss‚1.7623912502594035 /loss/FBt3/detection_loss‚2.9346781075000763 %loss/total_loss‚40.84616115279447 2loss/SRqqt4/assignment_loss‚2.3028030078858137 0loss/FBt4/assignment_loss‚0.4917383852570007 /loss/FRt1/detection_loss‚2.2813474969430403 0loss/SRqqt4/detection_loss‚2.542478329191605 /loss/FBt1/detection_loss‚2.4807360768318176 .loss/FBt4/detection_loss‚2.057953159014384 epoch‚7 0loss/FRt1/assignment_loss‚2.5248793946071104 0loss/FRt4/assignment_loss‚3.3062512786300093 2loss/SRqqt1/assignment_loss‚1.2262608967139386 _runtime‚7395.545850005 /loss/FRt2/detection_loss‚1.6074681142677687 .loss/FRt4/detection_loss‚3.118605845504337 % _timestamp‚1.7884553039914863e+09ŒÏÑ⛓ö#‚(:ˆ –ÓæÔÈŠÑÕ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 }i{E÷#‚(:º –ÓæÔßËÜØ disk.drbd1777.in‚0 disk.dm-0.in‚ 197.91796875 disk.rbd1.out‚ 3.6796875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.drbd1777.out‚0( proc.memory.percent‚17.5534725189209 disk.dm-1.out‚ 291.5546875 disk.sda1.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 948.70703125% disk./.usageGB‚1110.2620697021484 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 197.91796875 network.recv‚ 2.666274e+06 disk.sda.in‚ 0.01171875 disk.sda1.in‚0' disk.nvme0c0n1.out‚31989.6689453125 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2875.9609375 network.sent‚ 2.2846801e+07 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 291.5546875 cpu‚25.348139318311244) disk./.usagePercent‚74.51061906963668 disk.sda3.out‚ 291.5546875 disk.nvme0n1.in‚ 197.91796875" disk.dm-0.out‚31989.6689453125$ memory_percent‚94.20955181121826 disk.rbd2.in‚0! disk.rbd2.out‚239.48486328125% disk.nvme0n1.out‚31989.6689453125 proc.cpu.threads‚17}plò“ø#‚(:ˆ ¥ÓæÔǽÝÕ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— ZÈ:ù#‚(:¯ ¥ÓæÔÚ‡ºØ disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚292.0625 disk.rbd2.in‚0 network.sent‚ 2.2947584e+07 disk.sda1.in‚0 disk.nvme0n1.in‚ 198.67578125 disk.sda1.out‚0! disk.rbd2.out‚239.48486328125( disk.nvme0c0n1.out‚32090.12646484375 disk.dm-0.in‚ 198.67578125& disk.nvme0n1.out‚32090.12646484375$ memory_percent‚94.20046806335449' proc.memory.availableMB‚ 950.1953125" proc.memory.rssMB‚ 2875.9609375 disk.rbd0.in‚0 disk.rbd1.out‚ 3.6796875 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 network.recv‚ 2.67888e+06 disk.sda.out‚292.0625 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd3.in‚0( proc.memory.percent‚17.5534725189209 cpu‚25.34757447548029# disk./.usageGB‚1110.26167678833 disk.sda.in‚ 0.01171875 disk.sda3.out‚292.0625" disk.nvme0c0n1.in‚ 198.67578125# disk.dm-0.out‚32090.12646484375) disk./.usagePercent‚74.51059270086064 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.out‚0ǹ¤42ú#- _wandbruntime‚7431  _runtime‚7431Uè“Çû#Á & lr-AdamW/pg1‚2.370459248682578e-05 & lr-AdamW/pg2‚2.370459248682578e-05 trainer/global_step‚9549 % _timestamp‚1.7884553394917989e+09 _runtime‚7431.046201986 _step‚781&Ìü#Æ trainer/global_step‚9549 % _timestamp‚1.7884553394917989e+09 _runtime‚7431.046201986 _step‚781 & lr-AdamW/pg1‚2.370459248682578e-05 & lr-AdamW/pg2‚2.370459248682578e-05DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605488374537728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180104087 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7989177786880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126408022 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381392666624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834149376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318729929 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539914678170624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _runtime‚7431.047367743 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834149376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚782 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883590 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116ltF‘†ÿ#åO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605488374537728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180104087 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7989177786880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126408022 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381392666624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7989834149376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318729929 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539914678170624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _runtime‚7431.047367743 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7989834149376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/Z\(te requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚782 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554430476185088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547875118883590 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180104087 trainer/global_step‚9549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45529658 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554430476185088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138625844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613869791144448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310334464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126408482Ž[J2€$- _wandbruntime‚7431  _runtime‚7431 _(2ñ&$ëM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382249223680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126420714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45534182 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138639860 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚783 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382087710208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45534640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚7431.727134004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482814464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180142605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126421174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚539969599340544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613932298439168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 101942558725y]ö&‚$ðM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45534640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚7431.727134004 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562477482814464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180142605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126421174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚539969599340544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613932298439168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382249223680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126420714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45534182 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138639860 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚783 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382087710208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844/dÓ×2ƒ$- _wandbruntime‚7431  _runtime‚74310ÌÎY_„$Ù epoch‚7 2loss/SRqqt1/assignment_loss‚1.3616429278790427 1loss/SRqqt4/assignment_loss‚2.005302012215058 1loss/FBt4/assignment_loss‚0.22183393560616033 1loss/SRqqt2/detection_loss‚0.8994463580527476 0loss/SRqqt4/detection_loss‚2.405015065272649 /loss/FBt2/detection_loss‚0.5140777776078148 % _timestamp‚1.7884553401740398e+09 _step‚784 0loss/FRt2/assignment_loss‚1.5308918754259746 1loss/FBt1/assignment_loss‚0.24153632097877561 _runtime‚7431.728366937 0loss/FRt4/assignment_loss‚3.4702716903253035 2loss/SRqqt3/assignment_loss‚1.1140211433041514 /loss/FRt2/detection_loss‚1.5939621609800003 .loss/FRt3/detection_loss‚2.071359851360321 /loss/FBt1/detection_loss‚2.6677869319915772 /loss/FBt4/detection_loss‚2.5784748111452376 0loss/FRt1/assignment_loss‚2.5183490112021163 0loss/FRt3/assignment_loss‚2.5433983397483826 1loss/FBt2/assignment_loss‚0.12137225708307862 /loss/FRt1/detection_loss‚2.6080006948999457 2loss/SRqqt2/assignment_loss‚0.8616866365075111 0loss/FBt3/assignment_loss‚0.3733044488534298 1loss/SRqqt1/detection_loss‚1.5974700252215068 &loss/total_loss‚40.954520125912524 trainer/global_step‚9549 .loss/FRt4/detection_loss‚3.382110289551995 1loss/SRqqt3/detection_loss‚1.6591237522008127 /loss/FBt3/detection_loss‚2.6140818084989275*éTd…$Þ 0loss/FRt4/assignment_loss‚3.4702716903253035 2loss/SRqqt3/assignment_loss‚1.1140211433041514 /loss/FRt2/detection_loss‚1.5939621609800003 .loss/FRt3/detection_loss‚2.071359851360321 /loss/FBt1/detection_loss‚2.6677869319915772 /loss/FBt4/detection_loss‚2.5784748111452376 0loss/FRt1/assignment_loss‚2.5183490112021163 0loss/FRt3/assignment_loss‚2.5433983397483826 1loss/FBt2/assignment_loss‚0.12137225708307862 /loss/FRt1/detection_loss‚2.6080006948999457 2loss/SRqqt2/assignment_loss‚0.8616866365075111 0loss/FBt3/assignment_loss‚0.3733044488534298 1loss/SRqqt1/detection_loss‚1.5974700252215068 &loss/total_loss‚40.954520125912524 trainer/global_step‚9549 .loss/FRt4/detection_loss‚3.382110289551995 1loss/SRqqt3/detection_loss‚1.6591237522008127 /loss/FBt3/detection_loss‚2.6140818084989275 epoch‚7 2loss/SRqqt1/assignment_loss‚1.3616429278790427 1loss/SRqqt4/assignment_loss‚2.005302012215058 1loss/FBt4/assignment_loss‚0.22183393560616033 1loss/SRqqt2/detection_loss‚0.8994463580527476 0loss/SRqqt4/detection_loss‚2.405015065272649 /loss/FBt2/detection_loss‚0.5140777776078148 % _timestamp‚1.7884553401740398e+09 _step‚784 0loss/FRt2/assignment_loss‚1.5308918754259746 1loss/FBt1/assignment_loss‚0.24153632097877561 _runtime‚7431.728366937Yê"ž†$‚(:“ ´ÓæÔÅÍÃÕ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.623' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.724599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³\ÛC‡$‚(:¸ ´ÓæÔ¸Æï× disk.rbd2.in‚0' disk.nvme0c0n1.out‚32090.5947265625 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 292.78515625" proc.memory.rssMB‚ 2875.9609375% disk./.usageGB‚1110.2618522644043! disk.nvme0c0n1.in‚ 198.8828125 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875 network.sent‚ 2.296085e+07( proc.memory.availableMB‚ 948.86328125 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875" disk.dm-0.out‚32090.5947265625 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚25.34694722063946 disk.sda3.out‚ 292.78515625 disk.drbd1777.out‚0) disk./.usagePercent‚74.51060447720722 disk.nvme0n1.in‚ 198.8828125% disk.nvme0n1.out‚32090.5947265625 disk.sda2.out‚0( proc.memory.percent‚17.5534725189209 disk.sda.out‚ 292.78515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 198.8828125 disk.rbd3.out‚0$ memory_percent‚94.20859813690186 proc.cpu.threads‚17 network.recv‚ 2.697386e+06 disk.sda.in‚ 0.01171875! disk.rbd2.out‚241.22705078125PcäH“ˆ$‚(:ˆ ÃÓæÔ–ÉãÕ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.1818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F³Á=D‰$‚(:¹ ÃÓæÔâëºØ disk.rbd1.in‚0 disk.sda3.out‚ 293.5859375! disk.nvme0c0n1.in‚ 199.6015625) disk./.usagePercent‚74.51063417408119 disk.rbd0.out‚0 disk.nvme0n1.in‚ 199.6015625 disk.sda2.out‚0 disk.sda3.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0! disk.rbd2.out‚241.22705078125( disk.nvme0c0n1.out‚32188.69482421875 disk.dm-0.in‚ 199.6015625( proc.memory.availableMB‚ 949.89453125" proc.memory.rssMB‚ 2875.9609375 network.sent‚ 2.3068802e+07$ disk./.usageGB‚1110.262294769287 disk.sda.in‚ 0.01171875# disk.dm-0.out‚32188.69482421875 cpu‚25.346347657078475 network.recv‚ 2.704054e+06 disk.dm-1.out‚ 293.5859375 disk.rbd3.in‚0 disk.sda.out‚ 293.5859375 disk.rbd0.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚32188.69482421875$ memory_percent‚94.20230388641357 disk.rbd1.out‚ 3.6796875( proc.memory.percent‚17.5534725189209¢mxd2Š$- _wandbruntime‚7467  _runtime‚7467®ü¹ˆÉ‹$à ' lr-AdamW/pg2‚2.2821092940153058e-05 trainer/global_step‚9599 % _timestamp‚1.7884553758751414e+09 _runtime‚7467.429521024 _step‚785 ' lr-AdamW/pg1‚2.2821092940153058e-05Ú€ôÎŒ$È ' lr-AdamW/pg1‚2.2821092940153058e-05 ' lr-AdamW/pg2‚2.2821092940153058e-05 trainer/global_step‚9599 % _timestamp‚1.7884553758751414e+09 _runtime‚7467.429521024 _step‚785‘Y¬~2$- _wandbruntime‚7467  _runtime‚7467È<ôj Ž$àO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8030819955200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8030163592704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139326642 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163592704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557248128127488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139326642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45758643 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616995103636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273717248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181030139 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9599 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608570935361024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616995293298176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚565278948082688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45759106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030139 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127045407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181050177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8030819955200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8424357937152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚565278948082688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127045872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181030139 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9599 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608570935361024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616995293298176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚565278948082688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45759106 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030139 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376821 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127045407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181050177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8030819955200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8424357937152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚565278948082688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127045872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163592704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557248128127488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139326642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45758643 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616995103636992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0’éÁt2$- _wandbruntime‚7468  _runtime‚7468pGÍñ&‘$ëM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608632562158592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 % _timestamp‚1.7884553765436335e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446183936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81294932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9599 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚550716881316730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557304463163392 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617057799375360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8425213276672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617057605533184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127058142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45763672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139340658 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚565336120562688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557304463163392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45763210 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542714788829184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139340658 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448186 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 _runtime‚7468.098759753 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181068657 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8030983020544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ŠH™S’$ðM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45763672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139340658 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚565336120562688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557304463163392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45763210 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542714788829184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139340658 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448186 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 _runtime‚7468.098759753 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181068657 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8030983020544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608632562158592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 % _timestamp‚1.7884553765436335e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446183936 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557304481166336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81294932 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9599 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚550716881316730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557304463163392 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617057799375360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8425213276672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617057605533184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127058142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0“bžX>2“$- _wandbruntime‚7468  _runtime‚7468É %à]”$× 0loss/FRt4/assignment_loss‚2.7677912603725088 1loss/FBt3/assignment_loss‚0.05646095649626659 1loss/SRqqt3/detection_loss‚1.7094635114605938 $ _timestamp‚1.788455376545753e+09 0loss/FBt1/assignment_loss‚0.4054341135613088 1loss/FBt4/assignment_loss‚0.46382838244865826 /loss/FRt1/detection_loss‚2.2996384327004598 /loss/FRt4/detection_loss‚3.1399312913417816 /loss/FBt1/detection_loss‚2.4907084022249495 /loss/FBt2/detection_loss‚0.5457989154479552 /loss/FBt4/detection_loss‚2.6492360830307007 0loss/FRt3/assignment_loss‚3.1465543275294094 1loss/FBt2/assignment_loss‚0.17411102800102024 1loss/SRqqt4/detection_loss‚2.2096998393535614 epoch‚7 0loss/FRt2/assignment_loss‚1.4239964723587035 2loss/SRqqt2/assignment_loss‚0.9816449655968776 1loss/SRqqt2/detection_loss‚1.1672894625136485 0loss/SRqqt1/detection_loss‚1.559286477825334 _runtime‚7468.100075322 _step‚788 1loss/SRqqt1/assignment_loss‚1.332167104975651 /loss/FRt2/detection_loss‚1.6822323607280851 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.5846117758857352 .loss/FBt3/detection_loss‚3.218707104523977 %loss/total_loss‚41.64526004007863 1loss/SRqqt4/assignment_loss‚1.959347192198038 /loss/FRt3/detection_loss‚2.0888063823399468 0loss/FRt1/assignment_loss‚2.5885141971634655J!Wb•$Ü 0loss/FRt1/assignment_loss‚2.5885141971634655 0loss/FRt4/assignment_loss‚2.7677912603725088 1loss/FBt3/assignment_loss‚0.05646095649626659 1loss/SRqqt3/detection_loss‚1.7094635114605938 $ _timestamp‚1.788455376545753e+09 0loss/FBt1/assignment_loss‚0.4054341135613088 1loss/FBt4/assignment_loss‚0.46382838244865826 /loss/FRt1/detection_loss‚2.2996384327004598 /loss/FRt4/detection_loss‚3.1399312913417816 /loss/FBt1/detection_loss‚2.4907084022249495 /loss/FBt2/detection_loss‚0.5457989154479552 /loss/FBt4/detection_loss‚2.6492360830307007 0loss/FRt3/assignment_loss‚3.1465543275294094 1loss/FBt2/assignment_loss‚0.17411102800102024 1loss/SRqqt4/detection_loss‚2.2096998393535614 epoch‚7 0loss/FRt2/assignment_loss‚1.4239964723587035 2loss/SRqqt2/assignment_loss‚0.9816449655968776 1loss/SRqqt2/detection_loss‚1.1672894625136485 0loss/SRqqt1/detection_loss‚1.559286477825334 _runtime‚7468.100075322 _step‚788 1loss/SRqqt1/assignment_loss‚1.332167104975651 /loss/FRt2/detection_loss‚1.6822323607280851 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.5846117758857352 .loss/FBt3/detection_loss‚3.218707104523977 %loss/total_loss‚41.64526004007863 1loss/SRqqt4/assignment_loss‚1.959347192198038 /loss/FRt3/detection_loss‚2.0888063823399468”ܧyž–$‚(:“ ÒÓæÔÞž¨× gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.354' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.070800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^›W8—$‚(:­ ÒÓæÔôÀƒÙ disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 294.078125 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 294.078125 disk.rbd2.in‚0! disk.rbd2.out‚242.97314453125 network.recv‚ 2.715268e+06% disk.nvme0c0n1.out‚32190.05859375 disk.sda2.out‚0 disk.dm-1.out‚ 294.078125 disk.rbd1.out‚ 3.6796875$ memory_percent‚94.20688152313232) disk./.usagePercent‚74.51069459185928 proc.cpu.threads‚17 network.sent‚ 2.3078479e+07! disk.nvme0c0n1.in‚ 199.6015625 disk.sda.in‚ 0.01171875 disk.sda1.in‚0( proc.memory.percent‚17.5534725189209 cpu‚25.34578396419263 disk.nvme0n1.in‚ 199.6015625% disk./.usageGB‚1110.2631950378418 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 949.14453125" proc.memory.rssMB‚ 2875.9609375 disk.dm-0.out‚32190.05859375 disk.rbd0.out‚0# disk.nvme0n1.out‚32190.05859375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 199.6015625ŠÑÝÓ“˜$‚(:ˆ áÓæÔÑ»âÕ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌñûC™$‚(:¸ áÓæÔ©©ò× network.sent‚ 2.3190274e+07% disk./.usageGB‚1110.2593460083008 disk.sda.in‚ 0.01171875 disk.sda1.in‚0' disk.nvme0c0n1.out‚32288.9677734375 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0 disk.dm-0.in‚ 199.6015625" disk.dm-0.out‚32288.9677734375 disk.rbd1.in‚0! disk.rbd2.out‚244.54736328125 disk.nvme0n1.in‚ 199.6015625 disk.rbd3.in‚0$ memory_percent‚94.20809745788574 cpu‚25.345201197930233% disk.nvme0n1.out‚32288.9677734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 294.83984375' proc.memory.availableMB‚ 948.9453125 proc.cpu.threads‚17 disk.rbd0.in‚0" proc.memory.rssMB‚ 2875.9609375 disk.sda3.out‚ 294.83984375! disk.nvme0c0n1.in‚ 199.6015625 disk.rbd1.out‚ 3.6796875 disk.rbd3.out‚0( proc.memory.percent‚17.5534725189209 network.recv‚ 2.72918e+06) disk./.usagePercent‚74.51043628025722 disk.sda1.out‚0 disk.sda.out‚ 294.83984375 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0|Çd<š$‚(:… ðÓæÔî§ÚÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u]û@›$‚(:µ ðÓæÔ°ä Ù cpu‚25.34457001394656 network.sent‚ 2.319655e+07 disk.drbd1777.out‚0' proc.memory.availableMB‚ 948.3828125) disk./.usagePercent‚74.51044319246066# disk.dm-0.out‚32386.50732421875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.6015625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚32386.50732421875 disk.nvme0n1.in‚ 199.6015625 disk.dm-1.in‚ 0.01171875$ disk./.usageGB‚1110.259449005127 disk.sda3.out‚ 295.2734375! disk.rbd2.out‚244.66455078125 disk.rbd1.out‚ 3.6796875( proc.memory.percent‚17.5534725189209 disk.sda.out‚ 295.2734375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 295.2734375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 199.6015625 disk.sda2.in‚0 disk.rbd3.in‚0# memory_percent‚94.2115306854248 proc.cpu.threads‚17 network.recv‚ 2.733629e+06 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0& disk.nvme0n1.out‚32386.50732421875" proc.memory.rssMB‚ 2875.9609375ÀÜø–2œ$- _wandbruntime‚7503  _runtime‚7503Šo{åÇ$Á & lr-AdamW/pg2‚2.195203980538475e-05 trainer/global_step‚9649 % _timestamp‚1.7884554120007946e+09 _runtime‚7503.555235991 _step‚789 & lr-AdamW/pg1‚2.195203980538475e-05žMŸÌž$Æ & lr-AdamW/pg1‚2.195203980538475e-05 & lr-AdamW/pg2‚2.195203980538475e-05 trainer/global_step‚9649 % _timestamp‚1.7884554120007946e+09 _runtime‚7503.555235991 _step‚789•´«aÜ2Ÿ$- _wandbruntime‚7503  _runtime‚7503öt:Öæ' $àO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127683324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553447202618406 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120673486848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181976229 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788455412001614e+09 _runtime‚7503.556255792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8071149398528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45988160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140027444 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8467225182720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45988623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620120867342336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181956191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911465472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025NÙÁ£'¡$åO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127683324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553447202618406 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620120673486848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181976229 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788455412001614e+09 _runtime‚7503.556255792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8071149398528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45988160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140027444 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8467225182720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45988623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620120867342336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181956191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911465472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 DÚ5`1H>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0–À¼¯2¢$- _wandbruntime‚7504  _runtime‚7504£uµñ&£$ëM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45992719 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071968826368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127696051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140041460 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620183171745792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994709 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545459995357184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884554127176785e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974645 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚791 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036169 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚560122133108736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127696503 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944¹Ýö&¤$ðM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140041460 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994709 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚545459995357184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884554127176785e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974645 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚791 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036169 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚560122133108736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127696503 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45992719 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071968826368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127696051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140041460 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620183171745792—;ÏÖµ2¥$- _wandbruntime‚7504  _runtime‚7504¹H™v[¦$Õ 0loss/FBt2/assignment_loss‚0.2147778024255409 1loss/SRqqt1/detection_loss‚1.7294487249763573 0loss/SRqqt2/detection_loss‚1.078381240998323 1loss/SRqqt4/detection_loss‚3.0174909955576847 trainer/global_step‚9649 0loss/FRt2/assignment_loss‚1.4765625848914639 /loss/FRt4/assignment_loss‚3.183372974395752 2loss/SRqqt4/assignment_loss‚1.6903910668272721 0loss/FBt1/assignment_loss‚0.3728308189311065 2loss/SRqqt1/assignment_loss‚1.5833550360711182 2loss/SRqqt2/assignment_loss‚0.6663004292103534 0loss/FBt4/assignment_loss‚0.4991020699275065 /loss/FBt1/detection_loss‚3.0471769273281097 /loss/FRt1/detection_loss‚2.4982213183890942 /loss/FBt4/detection_loss‚2.7965629534287886 /loss/FBt2/detection_loss‚0.5708322841186563 /loss/FRt3/assignment_loss‚2.508467129789866 2loss/SRqqt3/assignment_loss‚1.0941046359948814 0loss/FBt3/assignment_loss‚0.1553726167185232 _step‚792 .loss/FRt3/detection_loss‚1.957017195625947 %loss/total_loss‚41.56492511542702 % _timestamp‚1.7884554127202072e+09 /loss/FRt2/detection_loss‚1.5569412794077035 /loss/FRt4/detection_loss‚2.6054944153875113 1loss/SRqqt3/detection_loss‚1.6410601905857523 /loss/FBt3/detection_loss‚2.5896253176033497 epoch‚7 _runtime‚7504.274557187 0loss/FRt1/assignment_loss‚3.0320351068363633‚Mr`§$Ú 0loss/FBt1/assignment_loss‚0.3728308189311065 2loss/SRqqt1/assignment_loss‚1.5833550360711182 2loss/SRqqt2/assignment_loss‚0.6663004292103534 0loss/FBt4/assignment_loss‚0.4991020699275065 /loss/FBt1/detection_loss‚3.0471769273281097 /loss/FRt1/detection_loss‚2.4982213183890942 /loss/FBt4/detection_loss‚2.7965629534287886 /loss/FBt2/detection_loss‚0.5708322841186563 /loss/FRt3/assignment_loss‚2.508467129789866 2loss/SRqqt3/assignment_loss‚1.0941046359948814 0loss/FBt3/assignment_loss‚0.1553726167185232 _step‚792 .loss/FRt3/detection_loss‚1.957017195625947 %loss/total_loss‚41.56492511542702 % _timestamp‚1.7884554127202072e+09 /loss/FRt2/detection_loss‚1.5569412794077035 /loss/FRt4/detection_loss‚2.6054944153875113 1loss/SRqqt3/detection_loss‚1.6410601905857523 /loss/FBt3/detection_loss‚2.5896253176033497 epoch‚7 _runtime‚7504.274557187 0loss/FRt1/assignment_loss‚3.0320351068363633 0loss/FBt2/assignment_loss‚0.2147778024255409 1loss/SRqqt1/detection_loss‚1.7294487249763573 0loss/SRqqt2/detection_loss‚1.078381240998323 1loss/SRqqt4/detection_loss‚3.0174909955576847 trainer/global_step‚9649 0loss/FRt2/assignment_loss‚1.4765625848914639 /loss/FRt4/assignment_loss‚3.183372974395752 2loss/SRqqt4/assignment_loss‚1.6903910668272721˜æ Èmž¨$‚(:“ ÿÓæÔãœÂÖ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.992' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.998399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÊ&J?©$‚(:´ ÿÓæÔˆÀì×! disk.rbd2.out‚246.23876953125 disk.nvme0n1.in‚ 199.6015625 disk.drbd1777.out‚0# disk.dm-0.out‚32387.07763671875 disk.rbd1.out‚ 3.6796875 disk.sda3.out‚ 295.90625 disk.sda.out‚ 295.90625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.percent‚17.5534725189209 disk.rbd0.out‚0 disk.dm-0.in‚ 199.6015625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.01171875$ memory_percent‚94.21088695526123 proc.cpu.threads‚17 network.recv‚ 2.736914e+06& disk.nvme0n1.out‚32387.07763671875( proc.memory.availableMB‚ 948.48828125% disk./.usageGB‚1110.2634353637695 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 199.6015625( disk.nvme0c0n1.out‚32387.07763671875 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 cpu‚25.343993947670405 network.sent‚ 2.3201597e+07 disk.rbd3.out‚0" proc.memory.rssMB‚ 2875.9609375) disk./.usagePercent‚74.51071072033393 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 295.90625Í$2$‘ª$‚(:† ŽÔæÔü˜†Ö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éƒI«$‚(:¾ ŽÔæÔ“€³Ø disk.sda3.in‚ 0.01171875 disk.drbd1777.in‚0 disk.sda2.in‚0$ memory_percent‚94.21265125274658 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 199.6015625# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.51070944029625 disk.rbd2.in‚0( disk.nvme0c0n1.out‚32484.57373046875 disk.dm-1.out‚ 296.54296875 disk.sda.out‚ 296.54296875! disk.nvme0c0n1.in‚ 199.6015625 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 296.54296875! disk.rbd2.out‚246.40673828125& disk.nvme0n1.out‚32484.57373046875# disk.dm-0.out‚32484.57373046875 network.recv‚ 2.74414e+06( proc.memory.availableMB‚ 948.19921875 network.sent‚ 2.3309647e+07 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 199.6015625 proc.cpu.threads‚17 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0 disk.sda2.out‚0 cpu‚25.350090185679047% disk./.usageGB‚1110.2634162902832 disk.rbd3.in‚0 disk.sda1.out‚0Ð!DM¬$‚(:’ ÔæÔ¼ËÕ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚167.198' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.439600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BêqA­$‚(:¶ Ôæ԰݉Ø! disk.dm-0.out‚32485.060546875 disk.rbd1.out‚ 3.6796875) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17) disk./.usagePercent‚74.51071968059763 disk.dm-0.in‚ 199.6015625 cpu‚25.359965917555538 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 network.sent‚ 2.3315225e+07 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 296.98828125 disk.rbd3.out‚0 network.recv‚ 2.748527e+06 disk.sda3.out‚ 296.98828125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 296.98828125# memory_percent‚94.2112922668457% disk./.usageGB‚1110.2635688781738$ disk.nvme0n1.out‚32485.060546875 disk.sda1.out‚0# proc.memory.rssMB‚ 2875.96484375! disk.nvme0c0n1.in‚ 199.6015625 disk.sda2.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 948.421875 disk.rbd0.out‚0! disk.rbd2.out‚247.98095703125& disk.nvme0c0n1.out‚32485.060546875 disk.nvme0n1.in‚ 199.6015625ŠFM›®$‚(:’ ¬ÔæÔ×Ô´Ö gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚96.375' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.275000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Я\HA¯$‚(:¶ ¬ÔæÔÜáåÝ( disk./.usagePercent‚74.5107120003716 disk.drbd1777.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881! disk.dm-0.out‚32582.583984375 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda.out‚ 297.40234375 disk.nvme0n1.in‚ 199.6015625 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 cpu‚25.36910112520026& disk.nvme0c0n1.out‚32582.583984375 disk.rbd1.out‚3.765625 disk.dm-0.in‚ 199.6015625 disk.dm-1.out‚ 297.40234375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 199.6015625$ memory_percent‚94.21145915985107 network.sent‚ 2.3321574e+07 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚32582.583984375( proc.memory.availableMB‚ 948.39453125 proc.cpu.threads‚17 network.recv‚ 2.753018e+06 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 297.40234375% disk./.usageGB‚1110.2634544372559 disk.rbd3.out‚0! disk.rbd2.out‚248.14501953125ZY%ˆ2°$- _wandbruntime‚7568  _runtime‚7568Ó„ê–±$ÈT JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚571275133537280 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128518943 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 140879304 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 182758607 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚563166571374592 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519100 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758607 % _timestamp‚1.7884554766865108e+09 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847270 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 trainer/global_step‚9671 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432414363648 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622704128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46317219 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432557841920 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚793 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚614927326212096 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46317374 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761662 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844lò™»S*²$ÍT YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚155 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 140879304 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 182758607 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚563166571374592 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128519100 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 182758607 % _timestamp‚1.7884554766865108e+09 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚563166571374592 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8078217847270 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 trainer/global_step‚9671 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚563166553371648 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82201724 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚623432414363648 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚571274622704128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚46317219 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚623432557841920 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _step‚793 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82201726 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚614927326212096 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚46317374 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚548397297483776 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8078708761662 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚571275133537280 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 128518943 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 323640042 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 % _timestamp‚1.7884554776160924e+09 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656021 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530118 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚556511155660582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚563203383400448 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚623470798702080 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8505572816384 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927289344 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚548432591243264 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 140890110 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚82208228 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚563203401403392 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚571312328692736 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚623470655223808 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚794 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚82208230 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚614965225885696 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 128530275Î`{wHµ$×R KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 128530275 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 182765909 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8108434459136 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 143478272 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @9DeviceStatsMonitorTY<ä.on_validation_batch_end/max_split_size‚-1 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 323640042 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 % _timestamp‚1.7884554776160924e+09 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656021 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128530118 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚155 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚556511155660582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚563203383400448 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚623470798702080 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8505572816384 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119538176 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8108927289344 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚548432591243264 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 140890110 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚82208228 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚563203401403392 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚571312328692736 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚623470655223808 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚794 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚82208230 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚614965225885696 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584šð7‰Ë‘¶$‚(:† »ÔæÔ…üëÕ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.768' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¢ãiA·$‚(:¶ »ÔæÔŸ´¯Ù) disk./.usagePercent‚74.51071609649215 disk.sda.out‚ 298.08203125 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 298.08203125 disk.rbd3.in‚0) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0c0n1.out‚32680.392578125 disk.rbd1.in‚0' proc.memory.availableMB‚ 948.3515625 cpu‚25.377646375246783 network.recv‚ 2.756799e+06 disk.rbd0.out‚0! disk.rbd2.out‚249.71923828125 network.sent‚ 2.3326934e+07! disk.nvme0c0n1.in‚ 199.6015625 disk.nvme0n1.in‚ 199.6015625! disk.dm-0.out‚32680.392578125 disk.rbd1.out‚3.828125# proc.memory.rssMB‚ 2875.96484375 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 298.08203125 disk.rbd0.in‚0 disk.drbd1777.in‚0$ disk./.usageGB‚1110.263515472412 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 199.6015625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd2.in‚0$ disk.nvme0n1.out‚32680.392578125 disk.sda2.out‚0$ memory_percent‚94.21172142028809 disk.rbd3.out‚0*@îå2¸$- _wandbruntime‚7581  _runtime‚7581q‰ZÄ`¹$è¥ @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5890042781829834 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4712856113910675 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.44243884086608887 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00239355000667274 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3724200129508972 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040515422821 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 $ _timestamp‚1.788455490130804e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25947701930999756 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.027419356629252434 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.53879314661026 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36296287178993225 jet/accuracy_1_of_7‚1 particle/accuracy_1_of_7‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5086749792098999 +particle/sensitivity‚0.3315969705581665 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11338444799184799 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5537634491920471 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2679106891155243 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5043710470199585 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9844093322753906 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.619940459728241 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982261061668396 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1034482792019844 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5076483488082886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5241479277610779 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19618624448776245 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5032167434692383 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010457848198711872 (particle/accuracy‚0.8543992042541504 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.09920634329319 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4987233579158783 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28011831641197205 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.594893217086792 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.573908805847168 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37361112236976624 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3697531223297119 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01750117354094982 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1463993787765503 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9758192300796509 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6033706665039062 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_5‚0.9527201652526855 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5916745066642761 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5334960222244263 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.538335919380188 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5581955909729004 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13627374172210693 *jet/accuracy_1_of_2‚0.7727389931678772 +jet/accuracy_2_of_3‚0.25858551263809204 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987770915031433 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15977780520915985 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46615684032440186 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1574830859899521 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17076075077056885 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23687702417373657 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14349016547203064 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09860224276781082 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3675568401813507 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5080912709236145 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9968350529670715 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025875916704535484 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2813698947429657 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3453703820705414 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4060077369213104 particle/accuracy_3_of_4‚1 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666507720947 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5655357837677002 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1507643312215805 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5833333134651184 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09125198423862457 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5383221507072449 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15940400958061218 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5459059476852417 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_2_of_2‚0.11930786818265915 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20574939250946045 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11302684247493744 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.1155593991279602 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2283557802438736 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5968766212463379 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚9671 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17086569964885712 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.45293232798576355 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4403049945831299 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43562743067741394 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5398322939872742 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9896037578582764 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3979646861553192 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893914461135864 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21969693899154663 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09028180688619614 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39642858505249023 *jet/accuracy_1_of_3‚0.8813940286636353 particle/accuracy_2_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16598640382289886 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41846486926078796 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5151295065879822 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15078315138816833 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9266304969787598 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4380702078342438 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04218946397304535 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871029257774353 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_4‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17273448407649994 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.653326690196991 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10668569058179855 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3455236256122589 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44764307141304016 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316873133182526 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865247666835785 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15555879473686218 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2227795273065567 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4908345937728882 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03801979869604111 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 +jet/accuracy_3_of_4‚0.06101609393954277 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16844499111175537 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820376038551331 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 _step‚795 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2807561159133911 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21669265627861023 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2002200037240982 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26365742087364197 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491228342056274 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49239420890808105 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9876959323883057 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793583869934082 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.576632022857666 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.486009806394577 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5405481457710266 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20095019042491913 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5393587350845337 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.199274942278862 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09468989074230194 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4210209846496582 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚42.42488517354425 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878416657447815 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6176281571388245 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20062264800071716 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022654224187135696 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 _runtime‚7581.687054038 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5660884976387024 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11816984415054321 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19172397255897522 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9734224081039429 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852360486984253 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1750205010175705 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.09523809701Épõ‹+22043 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35328996181488037 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49759817123413086 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19618624448776245 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46939578652381897 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 jet/accuracy_5_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24655170738697052 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_4‚0.4036862850189209 jet/accuracy_5_of_5‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5407007932662964 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5697656869888306 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_2_of_6‚0.43589746952056885 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08187135308980942 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5226398706436157 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894807934761047 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4883670508861542 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5277778506278992 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5212979912757874 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5537446141242981 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13499899208545685 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20819133520126343 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5405481457710266 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0042484113946557045 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140220165253 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028318881988525 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.625 particle/accuracy_1_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5076149702072144 particle/accuracy_1_of_2‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914035201072693 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5567755103111267 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20222222805023193 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.030890805646777153 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16534891724586487 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5467979907989502 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6936884522438049 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969764351844788 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5478395223617554 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6048610806465149 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4072868824005127 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21507957577705383 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_7‚0.5 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.649629533290863 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807366132736206 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9511393308639526 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02399848960340023 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23765434324741364 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5517524480819702 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.312440425157547 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8216540217399597 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2962748408317566 jet/accuracy_3_of_7‚0 epoch‚7 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02174859307706356 (particle/f_score‚0.41543900966644287 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5983294248580933 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35848212242126465 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05426774546504021 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47509583830833435 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.041319821029901505 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5363188982009888 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2862499952316284 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9874328970909119 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20274044573307037 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885522723197937 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5518765449523926 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008112874813377857 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416641831398 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5863724946975708 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6186144351959229 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39991000294685364 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027987534180283546 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23152732849121094 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038390371948480606 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2847709357738495‹ã®lÇMº$í¥ particle/accuracy_1_of_2‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914035201072693 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5567755103111267 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20222222805023193 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.030890805646777153 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16534891724586487 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5467979907989502 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6936884522438049 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969764351844788 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5478395223617554 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6048610806465149 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4072868824005127 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21507957577705383 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_7‚0.5 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.649629533290863 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807366132736206 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9511393308639526 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02399848960340023 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23765434324741364 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5517524480819702 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.312440425157547 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8216540217399597 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2962748408317566 jet/accuracy_3_of_7‚0 epoch‚7 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02174859307706356 (particle/f_score‚0.41543900966644287 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5983294248580933 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35848212242126465 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05426774546504021 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.47509583830833435 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.041319821029901505 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5363188982009888 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2862499952316284 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9874328970909119 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20274044573307037 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885522723197937 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5518765449523926 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008112874813377857 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416641831398 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5863724946975708 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6186144351959229 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39991000294685364 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027987534180283546 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23152732849121094 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038390371948480606 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2847709357738495 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5890042781829834 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4712856113910675 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.44243884086608887 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00239355000667274 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3724200129508972 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040515422821 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 $ _timestamp‚1.788455490130804e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25947701930999756 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.027419356629252434 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.53879314661026 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36296287178993225 jet/accuracy_1_of_7‚1 particle/accuracy_1_of_7‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5086749792098999 +particle/sensitivity‚0.3315969705581665 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11338444799184799 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5537634491920471 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2679106891155243 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5043710470199585 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9844093322753906 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2271825224161148 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.619940459728241 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982261061668396 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1034482792019844 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5076483488082886 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5241479277610779 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19618624448776245 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5032167434692383 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010457848198711872 (particle/accuracy‚0.8543992042541504 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.09920634329319 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4987233579158783 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28011831641197205 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.594893217086792 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.573908805847168 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37361112236976624 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3697531223297119 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01750117354094982 , Purity/1FRt1SRqqt2FBt/FBt_purity‚0.96875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1463993787765503 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9758192300796509 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6033706665039062 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_5‚0.9527201652526855 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5916745066642761 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5334960222244263 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.538335919380188 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5581955909729004 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13627374172210693 *jet/accuracy_1_of_2‚0.7727389931678772 +jet/accuracy_2_of_3‚0.25858551263809204 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987770915031433 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15977780520915985 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46615684032440186 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1574830859899521 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17076075077056885 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23687702417373657 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14349016547203064 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09860224276781082 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3675568401813507 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5080912709236145 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9968350529670715 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025875916704535484 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2813698947429657 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3453703820705414 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4060077369213104 particle/accuracy_3_of_4‚1 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666507720947 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5655357837677002 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1507643312215805 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5833333134651184 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09125198423862457 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5383221507072449 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15940400958061218 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5459059476852417 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_2_of_2‚0.11930786818265915 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20574939250946045 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1130268424¢·RË-E7493744 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.1155593991279602 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2283557802438736 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5968766212463379 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚9671 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17086569964885712 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.45293232798576355 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4403049945831299 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43562743067741394 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5398322939872742 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9896037578582764 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3979646861553192 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893914461135864 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21969693899154663 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09028180688619614 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39642858505249023 *jet/accuracy_1_of_3‚0.8813940286636353 particle/accuracy_2_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16598640382289886 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41846486926078796 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5151295065879822 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15078315138816833 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9266304969787598 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4380702078342438 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04218946397304535 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9871029257774353 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_4‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17273448407649994 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.653326690196991 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10668569058179855 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3455236256122589 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44764307141304016 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316873133182526 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865247666835785 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15555879473686218 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2227795273065567 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4908345937728882 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03801979869604111 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 +jet/accuracy_3_of_4‚0.06101609393954277 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16844499111175537 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_5‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820376038551331 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 _step‚795 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2807561159133911 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21669265627861023 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2002200037240982 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26365742087364197 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491228342056274 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49239420890808105 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9876959323883057 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793583869934082 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.576632022857666 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.486009806394577 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5405481457710266 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20095019042491913 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5393587350845337 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.199274942278862 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09468989074230194 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4210209846496582 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚42.42488517354425 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878416657447815 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6176281571388245 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20062264800071716 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022654224187135696 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 _runtime‚7581.687054038 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5660884976387024 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11816984415054321 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19172397255897522 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9734224081039429 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852360486984253 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1750205010175705 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35328996181488037 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49759817123413086 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19618624448776245 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46939578652381897 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 jet/accuracy_5_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24655170738697052 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_4‚0.4036862850189209 jet/accuracy_5_of_5‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5407007932662964 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5697656869888306 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_2_of_6‚0.43589746952056885 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08187135308980942 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5226398706436157 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894807934761047 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4883670508861542 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5277778506278992 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5212979912757874 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5537446141242981 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13499899208545685 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20819133520126343 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5405481457710266 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0042484113946557045 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140220165253 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028318881988525 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.625 particle/accuracy_1_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5076149702072144›ÛÅZ‘»$‚(:† ÊÔæÔ¤ÊÓÖ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜;/7¼$‚(:¬ ÊÔæÔª÷×) proc.memory.percent‚17.55349636077881! disk.dm-0.out‚33153.650390625 disk.rbd0.out‚0! disk.rbd2.out‚251.46142578125# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 199.6015625$ disk.nvme0n1.out‚33153.650390625% disk./.usageGB‚1110.2654037475586 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚25.38078607878521 network.sent‚ 2.3435382e+07 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 298.71875 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0( disk./.usagePercent‚74.5108428202216 disk.rbd1.out‚3.828125 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 199.6015625 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚91.29781723022461& disk.nvme0c0n1.out‚33153.650390625 network.recv‚ 2.763547e+06 disk.dm-0.in‚ 199.6015625 disk.sda.out‚ 298.71875 disk.sda1.out‚0 disk.sda3.out‚ 298.71875' proc.memory.availableMB‚ 1425.765625e)—9“½$‚(:ˆ ÙÔæÔÏÖÎÕ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚101.426' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ÃóM¾$‚(: ÙÔæÔ¸û£Ù disk.sda.in‚ 0.01171875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 199.6015625) proc.memory.percent‚17.55349636077881 disk.sda2.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1425.48828125 disk.dm-0.in‚ 199.6015625 disk.dm-1.out‚ 299.17578125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0 cpu‚25.380126551419565 network.recv‚ 2.77081e+06% disk./.usageGB‚1110.2635231018066 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.8603515625 disk.sda1.in‚0 network.sent‚ 2.3558882e+07) disk./.usagePercent‚74.51071660850722 disk.nvme0n1.in‚ 199.6015625 disk.rbd3.in‚0! disk.rbd2.out‚251.46142578125( disk.nvme0c0n1.out‚33276.14501953125# proc.memory.rssMB‚ 2875.96484375& disk.nvme0n1.out‚33276.14501953125 disk.rbd3.out‚0 disk.sda.out‚ 299.17578125 disk.sda3.out‚ 299.17578125$ memory_percent‚91.29951000213623 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0# disk.dm-0.out‚33276.14501953125ö>`2¿$- _wandbruntime‚7605  _runtime‚7605J9 :ÉÀ$à _step‚796 ' lr-AdamW/pg1‚2.1097614191891544e-05 ' lr-AdamW/pg2‚2.1097614191891544e-05 trainer/global_step‚9699 % _timestamp‚1.7884555140418243e+09 _runtime‚7605.596241075”? >ÎÁ$È _runtime‚7605.596241075 _step‚796 ' lr-AdamW/pg1‚2.1097614191891544e-05 ' lr-AdamW/pg2‚2.1097614191891544e-05 trainer/global_step‚9699 % _timestamp‚1.7884555140418243e+09œ…Ó€c2Â$- _wandbruntime‚7605  _runtime‚7605/ðy„ç'Ã$áO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8533496555520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82513318 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8533662276608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚565203723826688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8105854191912 _step‚797 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155174400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141409020 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565203705823744 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚550373821641728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8105199754156 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573338860998144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183364244 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9699 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55Ñ•ð"0373821641728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8105199754156 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573338860998144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183364244 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129016256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8533496555520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82513318 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8533662276608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚565203723826688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8105854191912 _step‚797 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155174400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141409020 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565203705823744 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9699 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788455514720296e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚565260076865536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625718158694912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8106016189120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129028860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚550428708732928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630978048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141423038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033464832 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028425 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Ã2ó&Ç$íM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630978048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141423038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033464832 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129028425 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚550428725772288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46507374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707843584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707843584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46506941 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625718348342784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382724 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚565260058862592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183382724 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚798 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8534517266944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚565260076865536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8136630978048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974602240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 7606.2750621 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9699 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788455514720296e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚565260076865536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625718158694912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8106016189120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82521486 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129028860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚550428708732928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46728611 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142109820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797342720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628781240226816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620204717221376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173111808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553119028169728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142109822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129650327 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270258 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561265717701048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46728611 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142109820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797342720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628781240226816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620204717221376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173111808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553119028169728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142109822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129650327 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184270258 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412548 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82929886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628843744323584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129663020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82929884 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚568077728807936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142123836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46733136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432614 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚7642.134928422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960408064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789456 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960408064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch¦àPœ_end/active.small_pool.allocated‚ 184308776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326412548 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616783872Õƒõ&×$ïM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82929884 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚568077728807936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142123836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46733136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432614 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚7642.134928422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960408064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789456 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960408064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326412548 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8177616783872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142123838 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129663468 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553173915260928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671213056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚568077710804992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628843938165760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412548 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82929886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628843744323584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129663020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¢Ȭ–2Ø$- _wandbruntime‚7642  _runtime‚7642v›U¡aÙ$Û 0loss/FRt1/assignment_loss‚2.8603065538406374 1loss/SRqqt2/detection_loss‚0.9975062805189662 /loss/FBt2/detection_loss‚0.4811054990915372 2loss/SRqqt3/assignment_loss‚1.4116494588267345 1loss/FBt4/assignment_loss‚0.22130116559565066 1loss/SRqqt1/detection_loss‚1.7538855023800382 1loss/FBt2/assignment_loss‚0.08615345918187103 1loss/FBt3/assignment_loss‚0.24299645373740966 1loss/SRqqt4/detection_loss‚2.2226918551482653 /loss/FBt4/detection_loss‚3.0877257108688356 _step‚803 /loss/FRt2/detection_loss‚1.5923480110036001 0loss/FRt3/assignment_loss‚2.8164417573383878 2loss/SRqqt4/assignment_loss‚1.8844204027401774 1loss/FBt1/assignment_loss‚0.16167614849109668 1loss/SRqqt3/detection_loss‚1.8064502159563394 trainer/global_step‚9749 _runtime‚7642.136504271 /loss/FBt3/detection_loss‚2.3228511260106015 epoch‚8 0loss/FRt2/assignment_loss‚1.5007314781347911 2loss/SRqqt1/assignment_loss‚1.5493720190983915 .loss/FRt1/detection_loss‚2.368021822869778 .loss/FRt3/detection_loss‚2.053828756724085 % _timestamp‚1.7884555505821972e+09 0loss/FRt4/assignment_loss‚3.3713224164901243 2loss/SRqqt2/assignment_loss‚0.9518059764746322 /loss/FBt1/detection_loss‚2.3101402148604393 /loss/FRt4/detection_loss‚3.0435267975253444 %loss/total_loss‚41.09825908290773PÝs+fÚ$à 1loss/SRqqt4/detection_loss‚2.2226918551482653 /loss/FBt4/detection_loss‚3.0877257108688356 _step‚803 /loss/FRt2/detection_loss‚1.5923480110036001 0loss/FRt3/assignment_loss‚2.8164417573383878 2loss/SRqqt4/assignment_loss‚1.8844204027401774 1loss/FBt1/assignment_loss‚0.16167614849109668 1loss/SRqqt3/detection_loss‚1.8064502159563394 trainer/global_step‚9749 _runtime‚7642.136504271 /loss/FBt3/detection_loss‚2.3228511260106015 epoch‚8 0loss/FRt2/assignment_loss‚1.5007314781347911 2loss/SRqqt1/assignment_loss‚1.5493720190983915 .loss/FRt1/detection_loss‚2.368021822869778 .loss/FRt3/detection_loss‚2.053828756724085 % _timestamp‚1.7884555505821972e+09 0loss/FRt4/assignment_loss‚3.3713224164901243 2loss/SRqqt2/assignment_loss‚0.9518059764746322 /loss/FBt1/detection_loss‚2.3101402148604393 /loss/FRt4/detection_loss‚3.0435267975253444 %loss/total_loss‚41.09825908290773 0loss/FRt1/assignment_loss‚2.8603065538406374 1loss/SRqqt2/detection_loss‚0.9975062805189662 /loss/FBt2/detection_loss‚0.4811054990915372 2loss/SRqqt3/assignment_loss‚1.4116494588267345 1loss/FBt4/assignment_loss‚0.22130116559565066 1loss/SRqqt1/detection_loss‚1.7538855023800382 1loss/FBt2/assignment_loss‚0.08615345918187103 1loss/FBt3/assignment_loss‚0.24299645373740966£«“û“Û$‚(:ˆ †ÕæÔýñÄÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ÿŠGÜ$‚(:¼ †ÕæÔµòŸØ network.sent‚ 2.3679027e+07 disk.nvme0n1.in‚ 199.6015625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.37816559449155 disk.sda.out‚ 300.76171875 disk.rbd2.in‚0$ memory_percent‚91.30001068115234' disk.nvme0c0n1.out‚33823.3251953125 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875! disk.rbd2.out‚254.87939453125 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.6015625" disk.dm-0.out‚33823.3251953125# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 network.recv‚ 2.786114e+06$ disk./.usageGB‚1110.263526916504 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.92333984375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 300.76171875 disk.sda1.in‚0& proc.memory.availableMB‚ 1425.40625) disk./.usagePercent‚74.51071686451475 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 300.76171875! disk.nvme0c0n1.in‚ 199.6015625% disk.nvme0n1.out‚33823.3251953125÷³Ê“Ý$‚(:ˆ •ÕæÔíü²× gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š—JÞ$‚(:¿ •ÕæÔÇËÎÙ disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 301.34765625) disk./.usagePercent‚74.51072044862023 disk.nvme0n1.in‚ 199.6015625 disk.sda3.out‚ 301.34765625 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 199.6015625' disk.nvme0c0n1.out‚33921.4267578125 disk.drbd1777.in‚0" disk.dm-0.out‚33921.4267578125) proc.memory.availableMB‚ 1425.37890625 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 disk.sda.out‚ 301.34765625% disk.nvme0n1.out‚33921.4267578125 network.sent‚ 2.3787186e+07 network.recv‚ 2.792911e+06% disk./.usageGB‚1110.2635803222656 disk.rbd0.out‚0! disk.rbd2.out‚254.87939453125 disk.drbd1777.out‚0 disk.dm-0.in‚ 199.6015625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda1.out‚0$ memory_percent‚91.30017757415771 cpu‚25.377516330354695 disk.sda3.in‚ 0.01171875Eü‡2ß$- _wandbruntime‚7677  _runtime‚7677A­6Æà$À _step‚804 & lr-AdamW/pg1‚1.943335468733822e-05 & lr-AdamW/pg2‚1.943335468733822e-05 trainer/global_step‚9799 % _timestamp‚1.7884555858613362e+09 _runtime‚ 7677.41573116 é:žËá$Å _step‚804 & lr-AdamW/pg1‚1.943335468733822e-05 & lr-AdamW/pg2‚1.943335468733822e-05 trainer/global_step‚9799 % _timestamp‚1.7884555858613362e+09 _runtime‚ 7677.41573116¤jè‹2â$- _wandbruntime‚7677  _runtime‚7677úé%'ç'ã$áO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚9799 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564051759568456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579056136494592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185196310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚570839009708544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46955508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130285202 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884555858620744e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631906831397376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7677.416692151 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130285626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619577292800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570839027711488 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46955508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623287230164480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130285202 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884555858620744e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peakäübñ‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631906831397376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7677.416692151 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130285626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619577292800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570839027711488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚9799 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872¥JÆ¢}2å$- _wandbruntime‚7678  _runtime‚7678‡½Ê’ñ&æ$ëM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328039400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46959554 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620280153600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188341305900 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620447958528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚806 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946213888 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130298264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46959978 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631969352800768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚7678.084708116 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚9799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884555865296602e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969161055744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130297838 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚623348904842240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚9799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83338286 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884555865296602e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969161055744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130297838 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚623348904842240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623348880902144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620280153600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188341305900 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620447958528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚806 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8217946213888 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130298264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46959978 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631969352800768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚7678.084708116 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 FGï$‚(:… ÂÕæÔ³ª¹Õ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚95.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êqaEð$‚(:º ÂÕæÔ¨ðèØ cpu‚25.375616421006466 network.recv‚ 2.808744e+06 disk.rbd1.out‚ 3.986328125 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.51084947641749 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 303.09765625! disk.rbd2.out‚258.29736328125 disk.sda2.out‚0 proc.memory.availableMB‚1425# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 199.60546875 disk.drbd1777.in‚0" memory_percent‚91.302490234375& disk.nvme0n1.out‚34119.13525390625 disk.dm-0.in‚ 199.60546875 network.sent‚ 2.3907808e+07% disk./.usageGB‚1110.2655029296875 disk.dm-1.out‚ 303.09765625 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 199.60546875 disk.sda2.in‚0 disk.sda.out‚ 303.09765625 disk.sda1.in‚0( disk.nvme0c0n1.out‚34119.13525390625# disk.dm-0.out‚34119.13525390625ÿBæ2ñ$- _wandbruntime‚7713  _runtime‚7713þ4´Éò$à _step‚808 ' lr-AdamW/pg1‚1.8623867625464364e-05 ' lr-AdamW/pg2‚1.8623867625464364e-05 trainer/global_step‚9849 % _timestamp‚1.7884556215878181e+09 _runtime‚7713.142216036¹Dd]Îó$È ' lr-AdamW/pg1‚1.8623867625464364e-05 ' lr-AdamW/pg2‚1.8623867625464364e-05 trainer/global_step‚9849 % _timestamp‚1.7884556215878181e+09 _runtime‚7713.142216036 _step‚808¨cÛhy2ô$- _wandbruntime‚7713  _runtime‚7713Ü`1xõ$áO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662475552256 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 `LDe’(C·Ö%viceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768954368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚809 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47182831 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032425969664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8258112591872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633782 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662634981888 % _timestamp‚1.7884556215886574e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112591872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448608256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130921794 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚7713.143211532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032242599936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566837801435864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768954368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚809 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47182831 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573656679653888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032425969664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573656661650944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8258112591872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633782 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8662634981888 % _timestamp‚1.7884556215886574e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112591872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581915448608256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130921794 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚7713.143211532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635032242599936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566837801435864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635094935728128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130934146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143525436 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626431441725440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8228522207344 trainer/global_step‚9849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47187462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884556224731398e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581972621088256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186160880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130934612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47187926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094739788800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573713014689792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186140816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8663494002688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581972621088256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚7714.028107605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚558664328316928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143525438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430:-ç(ø ù$ðM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635094935728128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_bõÐ$Hþatch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130934146 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143525436 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626431441725440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8228522207344 trainer/global_step‚9849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47187462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884556224731398e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581972621088256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186160880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130934612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47187926 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635094739788800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚573713014689792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186140816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8663494002688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581972621088256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚810 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚7714.028107605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚558664328316928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143525438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8258932019712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 Fz]ƒÎƒ%È ' lr-AdamW/pg1‚1.7829701670959564e-05 ' lr-AdamW/pg2‚1.7829701670959564e-05 trainer/global_step‚9899 % _timestamp‚1.7884556574856336e+09 _runtime‚7749.040014067 _step‚812¬~2„%- _wandbruntime‚7749  _runtime‚77493wç'…%áO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754760192 _step‚813 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638157784606208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826156 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7749.041079054 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561354630714368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269195549544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584773411991040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144212222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47412046 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638157588653568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144212222 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8299754760192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8705284722688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331280674 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111788 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584774086356480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131558964 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8705456735232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84146916 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584774086356480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84146918 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187048414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 % _timestamp‚1.7884556574864252e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47411619 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331260634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098397696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098397696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0õ¢v~û"†%æO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚7749.041079054 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561354630714368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269195549544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584773411991040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144212222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47412046 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638157588653568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144212222 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8299754760192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8705284722688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331280674 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111788 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584774086356480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131558964 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8705456735232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84146916 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584774086356480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84146918 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187048414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 % _timestamp‚1.7884556574864252e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47411619 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331260634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098397696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629452303930880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segmù+vÛñent.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098397696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754760192 _step‚813 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638157784606208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826156­zc 42‡%- _wandbruntime‚7749  _runtime‚7749±0Êñ&ˆ%ëM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258836480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47416150 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066868 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706149496320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576530684635136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131571234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚9899 _runtime‚7749.712498115 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131571677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086932 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569679563869004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8269357546752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258836480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187066868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8706313106944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884556581573703e+09 _step‚814 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220104164864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47416591 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638220291715584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569678892391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331313170 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521—ͨ|ö&‰%ðM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131571677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187086932 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569679563869004 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8269357546752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258836480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187066868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8706313106944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884556581573703e+09 _step‚814 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220104164864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47416591 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638220291715584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569678892391680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331313170 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258836480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47416150 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629513978608640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066868 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706149496320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576530684635136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576530666632192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131571234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚9899 _runtime‚7749.712498115®ø—EJ2Š%- _wandbruntime‚7749  _runtime‚7749²ŽfÐ^‹%Ø /loss/FRt4/assignment_loss‚3.772650567496695 1loss/SRqqt1/detection_loss‚1.6475107381741205 /loss/FBt3/detection_loss‚2.1572250525156655 0loss/FRt2/assignment_loss‚1.4570609582097906 /loss/FRt1/detection_loss‚2.5940139504467568 /loss/FRt2/detection_loss‚1.5895841233432293 1loss/SRqqt2/detection_loss‚0.9112452405026635 1loss/SRqqt4/detection_loss‚2.3199081130516834 /loss/FBt4/detection_loss‚2.9221760290009633 # _timestamp‚1.78845565815932e+09 _runtime‚7749.713957865 1loss/FBt4/assignment_loss‚0.28233894269214943 /loss/FBt2/detection_loss‚0.7601721181200622 2loss/SRqqt3/assignment_loss‚1.2824940308540438 1loss/FBt3/assignment_loss‚0.22439497870315428 trainer/global_step‚9899 2loss/SRqqt1/assignment_loss‚1.2395841648802162 1loss/SRqqt4/assignment_loss‚1.569688920982373 1loss/SRqqt3/detection_loss‚1.9243032132896285 /loss/FBt1/detection_loss‚2.1130258664488792 %loss/total_loss‚40.03583098742031 1loss/FBt1/assignment_loss‚0.12087211950345278 1loss/FBt2/assignment_loss‚0.08329861183549091 epoch‚8 0loss/FRt1/assignment_loss‚2.2733909560412897 /loss/FRt3/assignment_loss‚3.257753510028124 /loss/FRt4/detection_loss‚2.7529107085088405 2loss/SRqqt2/assignment_loss‚0.8188514479882764 /loss/FRt3/detection_loss‚1.9613766248027638 _step‚815M3KcŒ%Ý 2loss/SRqqt1/assignment_loss‚1.2395841648802162 1loss/SRqqt4/assignment_loss‚1.569688920982373 1loss/SRqqt3/detection_loss‚1.9243032132896285 /loss/FBt1/detection_loss‚2.1130258664488792 %loss/total_loss‚40.03583098742031 1loss/FBt1/assignment_loss‚0.12087211950345278 1loss/FBt2/assignment_loss‚0.08329861183549091 epoch‚8 0loss/FRt1/assignment_loss‚2.2733909560412897 /loss/FRt3/assignment_loss‚3.257753510028124 /loss/FRt4/detection_loss‚2.7529107085088405 2loss/SRqqt2/assignment_loss‚0.8188514479882764 /loss/FRt3/detection_loss‚1.9613766248027638 _step‚815 /loss/FRt4/assignment_loss‚3.772650567496695 1loss/SRqqt1/detection_loss‚1.6475107381741205 /loss/FBt3/detection_loss‚2.1572250525156655 0loss/FRt2/assignment_loss‚1.4570609582097906 /loss/FRt1/detection_loss‚2.5940139504467568 /loss/FRt2/detection_loss‚1.5895841233432293 1loss/SRqqt2/detection_loss‚0.9112452405026635 1loss/SRqqt4/detection_loss‚2.3199081130516834 /loss/FBt4/detection_loss‚2.9221760290009633 # _timestamp‚1.78845565815932e+09 _runtime‚7749.713957865 1loss/FBt4/assignment_loss‚0.28233894269214943 /loss/FBt2/detection_loss‚0.7601721181200622 2loss/SRqqt3/assignment_loss‚1.2824940308540438 1loss/FBt3/assignment_loss‚0.22439497870315428 trainer/global_step‚9899¯þ¶puž%‚(:“ ïÕæÔȼßÕ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.697000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܘ¼a2Ž%‚(:§ ïÕæÔŒóó×) proc.memory.percent‚17.55349636077881 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.out‚305 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875" disk.dm-0.out‚34315.6201171875 disk.rbd1.out‚ 3.986328125# proc.memory.rssMB‚ 2875.96484375 cpu‚25.373739326042493% disk.nvme0n1.out‚34315.6201171875 disk.rbd0.in‚0! disk.rbd2.out‚261.60986328125" disk.nvme0c0n1.in‚ 199.60546875 network.sent‚ 2.4026453e+07 disk.drbd1777.out‚0% disk./.usageGB‚1110.2649383544922 disk.rbd3.in‚0$ memory_percent‚91.30008220672607) proc.memory.availableMB‚ 1425.39453125 disk.nvme0n1.in‚ 199.60546875 disk.dm-0.in‚ 199.60546875 disk.dm-1.out‚305 disk.rbd1.in‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.823071e+06) disk./.usagePercent‚74.51081158730241 disk.sda2.out‚0 disk.sda3.out‚305' disk.nvme0c0n1.out‚34315.6201171875š÷>Ž%‚(:ƒ þÕæÔý™ÇÕ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚103.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê“ÍÁ5%‚(:ª þÕæÔª–¢Ù disk.rbd0.out‚0' proc.memory.availableMB‚ 951.1640625) proc.memory.percent‚17.55349636077881 network.recv‚ 2.830089e+06 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875% disk.nvme0n1.out‚34315.9755859375" disk.dm-0.out‚34315.9755859375 disk.dm-1.out‚305.5( disk./.usagePercent‚74.5105975650038 disk.sda3.out‚305.5 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 199.60546875 disk.drbd1777.out‚0 disk.sda2.in‚0% disk./.usageGB‚1110.2617492675781 disk.rbd2.in‚0 disk.nvme0n1.in‚ 199.60546875 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.373139290921017 network.sent‚ 2.4135239e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.01171875! disk.rbd2.out‚261.77783203125# proc.memory.rssMB‚ 2875.96484375 disk.rbd1.in‚0 disk.sda.out‚305.5' disk.nvme0c0n1.out‚34315.9755859375 disk.drbd1777.in‚0$ memory_percent‚94.19455528259277 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-0.in‚ 199.60546875 disk.rbd1.out‚ 3.986328125>iÚj2‘%- _wandbruntime‚7784  _runtime‚7784[kEòÉ’%à ' lr-AdamW/pg1‚1.7051022326818366e-05 ' lr-AdamW/pg2‚1.7051022326818366e-05 trainer/global_step‚9949 % _timestamp‚1.7884556934196758e+09 _runtime‚7784.974091012 _step‚816ãõ;Γ%È ' lr-AdamW/pg2‚1.7051022326818366e-05 trainer/global_step‚9949 % _timestamp‚1.7884556934196758e+09 _runtime‚7784.974091012 _step‚816 ' lr-AdamW/pg1‚1.7051022326818366e-05°¢ðqµ2”%- _wandbruntime‚7784  _runtime‚7784€©ËÇ•%àO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740566016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632534840814080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632534864754176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132195424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 7784.97494477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587632049739264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641283346505216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283154746880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748313932800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884556934204168e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚817 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579291965535744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚587632724104704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0·a/µë'–%åO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632534864754176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132195424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 7784.97494477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587632049739264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641283346505216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚9949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283154746880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748313932800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884556934204168e+09 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _step‚817 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579291965535744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚587632724104704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740566016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632534840814080±Å¥‚’—%‚(:‡ ÖæÔDZÕÕ gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚89.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú&éL˜%‚(:Á ÖæÔµüÀÙ) disk./.usagePercent‚74.51086202078666 disk.rbd3.in‚0! disk.rbd2.out‚263.35205078125 disk.drbd1777.out‚0# disk.dm-0.out‚34413.77490234375 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.nvme0n1.in‚ 199.60546875$ memory_percent‚94.19615268707275 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 199.60546875( disk.nvme0c0n1.out‚34413.77490234375 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.60546875 disk.sda2.out‚0 disk.sda3.out‚ 305.9765625 network.sent‚ 2.4140762e+07 network.recv‚ 2.834241e+06 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚34413.77490234375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 cpu‚25.372517109919414 disk.rbd1.out‚ 3.986328125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 305.9765625( proc.memory.availableMB‚ 950.90234375) proc.memory.percent‚17.55349636077881% disk./.usageGB‚1110.2656898498535 disk.rbd1.in‚0 disk.sda.out‚ 305.9765625ð]Ù2™%- _wandbruntime‚7785  _runtime‚7785©¯ð&š%êM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132208074 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896584704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 _runtime‚ 7785.65073996 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332919956 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84563486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8341560007168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749336691200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749166789120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340903631360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222205952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144927038 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84563486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8341560007168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749336691200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749166789120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340903631360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚587689222205952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144927038 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188012984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132208074 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587689896584704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 _runtime‚ 7785.65073996 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949²l–ö2œ%- _wandbruntime‚7785  _runtime‚7785Iä–]%× 0loss/FRt4/assignment_loss‚4.0856528878211975 1loss/SRqqt2/assignment_loss‚0.993193787910665 1loss/SRqqt4/assignment_loss‚2.381304689816066 3loss/FBt1/assignment_loss‚0.0012829819715989288 0loss/FBt3/assignment_loss‚0.3744068540011843 /loss/FBt3/detection_loss‚2.4295703834957547 %loss/total_loss‚43.54057565953224 trainer/global_step‚9949 0loss/FRt1/assignment_loss‚2.4509500047055686 0loss/FRt3/assignment_loss‚3.1117906925971046 1loss/SRqqt1/assignment_loss‚1.421992875569101 1loss/SRqqt3/detection_loss‚1.7641736209392547 _runtime‚7785.651830064 _step‚819 0loss/FRt2/assignment_loss‚1.7841453194618224 2loss/SRqqt3/assignment_loss‚1.2701020351797343 /loss/FRt1/detection_loss‚2.3538706646096417 .loss/FRt4/detection_loss‚3.157090250651042 1loss/SRqqt1/detection_loss‚1.5600068817979522 1loss/SRqqt4/detection_loss‚2.1730275682040623 /loss/FBt2/detection_loss‚0.7937450851922683 .loss/FBt4/detection_loss‚3.076880633831024 1loss/FBt2/assignment_loss‚0.10915539394132168 $ _timestamp‚1.788455694097535e+09 /loss/FRt2/detection_loss‚1.6004271940751509 1loss/FBt4/assignment_loss‚0.02493880786933005 .loss/FRt3/detection_loss‚2.186592426739241 /loss/FBt1/detection_loss‚3.2979107995827994 1loss/SRqqt2/detection_loss‚1.1383638195693493 epoch‚8FÊTôbž%Ü 1loss/FBt4/assignment_loss‚0.02493880786933005 .loss/FRt3/detection_loss‚2.186592426739241 /loss/FBt1/detection_loss‚3.2979107995827994 1loss/SRqqt2/detection_loss‚1.1383638195693493 epoch‚8 0loss/FRt4/assignment_loss‚4.0856528878211975 1loss/SRqqt2/assignment_loss‚0.993193787910665 1loss/SRqqt4/assignment_loss‚2.381304689816066 3loss/FBt1/assignment_loss‚0.0012829819715989288 0loss/FBt3/assignment_loss‚0.3744068540011843 /loss/FBt3/detection_loss‚2.4295703834957547 %loss/total_loss‚43.54057565953224 trainer/global_step‚9949 0loss/FRt1/assignment_loss‚2.4509500047055686 0loss/FRt3/assignment_loss‚3.1117906925971046 1loss/SRqqt1/assignment_loss‚1.421992875569101 1loss/SRqqt3/detection_loss‚1.7641736209392547 _runtime‚7785.651830064 _step‚819 0loss/FRt2/assignment_loss‚1.7841453194618224 2loss/SRqqt3/assignment_loss‚1.2701020351797343 /loss/FRt1/detection_loss‚2.3538706646096417 .loss/FRt4/detection_loss‚3.157090250651042 1loss/SRqqt1/detection_loss‚1.5600068817979522 1loss/SRqqt4/detection_loss‚2.1730275682040623 /loss/FBt2/detection_loss‚0.7937450851922683 .loss/FBt4/detection_loss‚3.076880633831024 1loss/FBt2/assignment_loss‚0.10915539394132168 $ _timestamp‚1.788455694097535e+09 /loss/FRt2/detection_loss‚1.6004271940751509³ž ªÕžŸ%‚(:“ œÖæÔôûÜÕ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.630200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$s*N %‚(:à œÖæÔÓÉØ disk.rbd0.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚34511.30224609375 network.sent‚ 2.423828e+07 disk.rbd3.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 199.60546875# proc.memory.rssMB‚ 2875.96484375( disk.nvme0c0n1.out‚34511.30224609375 disk.nvme0n1.in‚ 199.60546875( proc.memory.availableMB‚ 950.58203125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 2.841155e+06) disk./.usagePercent‚74.51085971671885 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.dm-0.in‚ 199.60546875% disk./.usageGB‚1110.2656555175781! disk.rbd2.out‚265.02783203125& disk.nvme0n1.out‚34511.30224609375 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.371873250076714 disk.sda3.in‚ 0.01171875 disk.rbd1.in‚0 disk.sda.out‚ 306.55078125 disk.dm-1.out‚ 306.55078125$ memory_percent‚94.19810771942139) proc.memory.percent‚17.55349636077881 disk.rbd1.out‚ 3.986328125 disk.sda3.out‚ 306.55078125ÚTí‹‘¡%‚(:† «ÖæÔóÓëÕ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚159.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òbH¢%‚(:½ «ÖæÔ–Ý©Ø disk.rbd3.out‚0 disk.sda.out‚ 306.9921875# memory_percent‚94.1960096359253( proc.memory.availableMB‚ 950.92578125% disk./.usageGB‚1110.2617530822754 disk.sda1.in‚0' disk.nvme0c0n1.out‚34511.7802734375 disk.nvme0n1.in‚ 199.60546875 disk.drbd1777.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.55349636077881 disk.sda3.out‚ 306.9921875 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.60546875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 199.60546875) disk./.usagePercent‚74.51059782101133 disk.rbd0.out‚0% disk.nvme0n1.out‚34511.7802734375 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 306.9921875# proc.memory.rssMB‚ 2875.96484375 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0! disk.rbd2.out‚265.02783203125" disk.dm-0.out‚34511.7802734375 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 cpu‚25.371259086488134 network.sent‚ 2.4335366e+07 network.recv‚ 2.847317e+06#Ã5x2£%- _wandbruntime‚7820  _runtime‚7820<×7ɤ%à ' lr-AdamW/pg2‚1.6287991868649027e-05 trainer/global_step‚9999 % _timestamp‚1.7884557288277974e+09 _runtime‚7820.382195199 _step‚820 ' lr-AdamW/pg1‚1.6287991868649027e-05’ˆ¸Î¥%È ' lr-AdamW/pg2‚1.6287991868649027e-05 trainer/global_step‚9999 % _timestamp‚1.7884557288277974e+09 _runtime‚7820.382195199 _step‚820 ' lr-AdamW/pg1‚1.6287991868649027e-05´¿Zž§2¦%- _wandbruntime‚7820  _runtime‚7820÷™N"ç'§%áO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566845060809728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644408593317888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791215620608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8381070009344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408774590464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361852928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566845043770368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47868567 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560972 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47869009 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8791372953088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145613820 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132832283 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920556 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726371840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7820.383250507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚821 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590491361852928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582109635481088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884557288286045e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496U[ì'¨%æO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47868567 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560972 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47869009 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8791372953088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145613820 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132832283 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920556 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726371840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7820.383250507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚821 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590491361852928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582109635481088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884557288286045e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566845060809728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644408593317888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791215620608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8381070009344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644408774590464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361852928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566845043770368µŠýs2©%- _wandbruntime‚7821  _runtime‚7821A2®èª%ëM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47873084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_bfô atch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590548534332928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792238052352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132845422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8381889437184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132844968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884557294927864e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590548534332928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792080733184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188939036 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644471290427392 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145627836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47873536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7821.047824823 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145627838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8382545812992xJJÒö&«%ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582165970516992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8381889437184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132844968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334546808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884557294927864e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590548534332928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792080733184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188939036 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644471290427392 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145627836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47873536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7821.047824823 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145627838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8382545812992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47873084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590548534332928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8792238052352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132845422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582165988519936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635679052375040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¶dØÑ2¬%- _wandbruntime‚7821  _runtime‚7821$À‘PY­%Ó 1loss/FBt2/assignment_loss‚0.11674562424516861 epoch‚8 0loss/FRt1/assignment_loss‚2.4507907100699167 /loss/FRt4/assignment_loss‚3.495067729669459 /loss/FRt3/detection_loss‚2.1034009451312676 0loss/FRt2/assignment_loss‚1.7976989471912384 1loss/FBt1/assignment_loss‚0.12557003942492884 .loss/FRt4/detection_loss‚3.547722006545347 0loss/SRqqt3/detection_loss‚1.588343988612013 /loss/FBt3/detection_loss‚2.8130539134144783 0loss/FRt3/assignment_loss‚3.0157585250479833 2loss/SRqqt1/assignment_loss‚1.4802507172442145 /loss/FBt3/assignment_loss‚0.733806811040111 1loss/FBt4/assignment_loss‚0.40373890940585017 trainer/global_step‚9999 _runtime‚7821.048988586 % _timestamp‚1.7884557294946547e+09 _step‚823 1loss/SRqqt4/detection_loss‚2.4521895972165195 .loss/FBt4/detection_loss‚3.058877189954122 %loss/total_loss‚43.57969211330484 2loss/SRqqt2/assignment_loss‚1.1018739793604861 .loss/FRt1/detection_loss‚2.391333477740938 /loss/FRt2/detection_loss‚1.7968592187017203 2loss/SRqqt3/assignment_loss‚1.4121653562376522 1loss/SRqqt4/assignment_loss‚1.388512765593601 1loss/SRqqt1/detection_loss‚1.7964483062426249 1loss/SRqqt2/detection_loss‚1.0193652735195227 .loss/FBt1/detection_loss‚2.729655921459198 .loss/FBt2/detection_loss‚0.760462160236479B[Õ4^®%Ø .loss/FBt2/detection_loss‚0.760462160236479 1loss/FBt2/assignment_loss‚0.11674562424516861 epoch‚8 0loss/FRt1/assignment_loss‚2.4507907100699167 /loss/FRt4/assignment_loss‚3.495067729669459 /loss/FRt3/detection_loss‚2.1034009451312676 0loss/FRt2/assignment_loss‚1.7976989471912384 1loss/FBt1/assignment_loss‚0.12557003942492884 .loss/FRt4/detection_loss‚3.547722006545347 0loss/SRqqt3/detection_loss‚1.588343988612013 /loss/FBt3/detection_loss‚2.8130539134144783 0loss/FRt3/assignment_loss‚3.0157585250479833 2loss/SRqqt1/assignment_loss‚1.4802507172442145 /loss/FBt3/assignment_loss‚0.733806811040111 1loss/FBt4/assignment_loss‚0.40373890940585017 trainer/global_step‚9999 _runtime‚7821.048988586 % _timestamp‚1.7884557294946547e+09 _step‚823 1loss/SRqqt4/detection_loss‚2.4521895972165195 .loss/FBt4/detection_loss‚3.058877189954122 %loss/total_loss‚43.57969211330484 2loss/SRqqt2/assignment_loss‚1.1018739793604861 .loss/FRt1/detection_loss‚2.391333477740938 /loss/FRt2/detection_loss‚1.7968592187017203 2loss/SRqqt3/assignment_loss‚1.4121653562376522 1loss/SRqqt4/assignment_loss‚1.388512765593601 1loss/SRqqt1/detection_loss‚1.7964483062426249 1loss/SRqqt2/detection_loss‚1.0193652735195227 .loss/FBt1/detection_loss‚2.729655921459198·ÁZ빓¯%‚(:ˆ ºÖæÔþ–ÊÕ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¿t"G°%‚(:¼ ºÖæÔ®ÁåØ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 307.828125& disk.nvme0n1.out‚34612.18310546875 cpu‚25.37064261392499) disk./.usagePercent‚74.51079238673735 disk.rbd1.in‚0 disk.sda.out‚ 307.828125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚94.19589042663574' proc.memory.availableMB‚ 950.9453125# disk.dm-0.out‚34612.18310546875 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 3.986328125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 199.60546875# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17% disk./.usageGB‚1110.2646522521973 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.60546875 network.sent‚ 2.4341631e+07 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875! disk.rbd2.out‚266.83251953125 disk.dm-1.out‚ 307.828125 network.recv‚ 2.851867e+06 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 199.60546875( disk.nvme0c0n1.out‚34612.18310546875J¦yÜž±%‚(:“ ÉÖæÔÛòóÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.501' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.900200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”°u J²%‚(:¿ ÉÖæÔ¹õ™Ø disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚34710.48974609375& disk.nvme0n1.out‚34710.48974609375 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 disk.rbd0.out‚0 network.sent‚ 2.4449678e+07 disk.sda3.out‚ 308.1953125 disk.nvme0n1.in‚ 199.60546875 disk.drbd1777.out‚0' proc.memory.availableMB‚ 950.9296875) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 disk.drbd1777.in‚0# disk.dm-0.out‚34710.48974609375 disk.rbd3.in‚0 disk.sda.out‚ 308.1953125$ memory_percent‚94.19598579406738 disk.dm-1.out‚ 308.1953125! disk.rbd2.out‚266.83251953125" disk.nvme0c0n1.in‚ 199.60546875 network.recv‚ 2.858452e+06) disk./.usagePercent‚74.51079776289558 disk.sda1.out‚0 cpu‚25.37003539992386 disk.sda1.in‚0 disk.dm-0.in‚ 199.60546875% disk./.usageGB‚1110.2647323608398 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd1.out‚ 3.986328125Q+Ì2³%- _wandbruntime‚7855  _runtime‚7855ñl¿MÊ´%Ä _step‚824 ' lr-AdamW/pg1‚1.5540769310855322e-05 ' lr-AdamW/pg2‚1.5540769310855322e-05 trainer/global_step‚10049 % _timestamp‚1.7884557644372547e+09 _runtime‚7855.991659979o2Qϵ%É _runtime‚7855.991659979 _step‚824 ' lr-AdamW/pg1‚1.5540769310855322e-05 ' lr-AdamW/pg2‚1.5540769310855322e-05 trainer/global_step‚10049 % _timestamp‚1.7884557644372547e+09¸ͲgB2¶%- _wandbruntime‚7855  _runtime‚7855c¾÷·%âO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999601152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161230 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325235712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48096466 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577981297428380 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884557644379284e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999601152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8422055815168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146314620 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534203965952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48096924 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189846608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚7855.992525039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584927287423488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834289385472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189826570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638699938520576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133468582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569590267337728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8834448815104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577981968905496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytÔÒ–ïñes.all.current‚ 183369728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712177664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8422712177664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146314622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133469042 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012'.Iþí'¸%çO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8422055815168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146314620 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚647534203965952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48096924 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584927269420544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189846608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚7855.992525039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584927287423488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834289385472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189826570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638699938520576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133468582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569590267337728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8834448815104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577981968905496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422712177664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8422712177664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146314622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133469042 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚593349999601152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189846608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161230 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚593349325235712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48096466 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577981297428380 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055815168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884557644379284e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999601152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952¹XJÏ2¹%- _wandbruntime‚7856  _runtime‚7856~Cê/ò&º%ìM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8835305763328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10049 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584983640462336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48101411 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48100958 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚7856.669387672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8423531618816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596694887936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8835129569792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647596895021568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584983640462336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884557651143649e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133481242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚826 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133481697 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚593406497702400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189865088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189845024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638761565318144 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423531618816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872}qðx÷&»%ñM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189865088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189845024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638761565318144 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423531618816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8835305763328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10049 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584983640462336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48101411 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48100958 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚7856.669387672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8423531618816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚647596694887936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8835129569792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647596895021568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584983640462336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146328636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884557651143649e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133481242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚826 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133481697 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚593406497702400ºÄ›á2¼%- _wandbruntime‚7856  _runtime‚7856?hŽf¤½%Ö /loss/FRt4/detection_loss‚3.0668894410133363 .loss/FBt1/detection_loss‚2.442196794918605 %loss/total_loss‚41.16766767511908 0loss/FRt1/assignment_loss‚2.3438741291562715 0loss/FRt3/assignment_loss‚3.0806687144856704 1loss/FBt1/assignment_loss‚0.33151140370111015 1loss/SRqqt2/detection_loss‚0.8559807653984298 _runtime‚7856.670760228 /loss/FRt4/assignment_loss‚3.7941305732727Ã` ¸05 0loss/FBt4/assignment_loss‚0.2103701982011374 .loss/FBt4/detection_loss‚2.323973224713252 trainer/global_step‚10049 0loss/FRt2/assignment_loss‚1.2461338593409612 1loss/SRqqt1/detection_loss‚1.7244369346234534 2loss/SRqqt1/assignment_loss‚1.6354930894794288 2loss/SRqqt2/assignment_loss‚1.2120563059613325 2loss/SRqqt4/assignment_loss‚1.6194356331587627 1loss/FBt3/assignment_loss‚0.46313649956136943 /loss/FRt3/detection_loss‚2.1272475084191873 epoch‚8 _step‚827 2loss/SRqqt3/assignment_loss‚1.2713911119582397 -loss/FRt2/detection_loss‚1.37122929124878 /loss/FBt2/detection_loss‚0.5167783314054184 % _timestamp‚1.7884557651164653e+09 0loss/FBt2/assignment_loss‚0.2408094020669846 /loss/FRt1/detection_loss‚2.3133237302924194 /loss/FBt3/detection_loss‚3.0990281820297243 1loss/SRqqt3/detection_loss‚1.5090080985267247 0loss/SRqqt4/detection_loss‚2.368564452185775[¥Za¾%Û 0loss/FBt2/assignment_loss‚0.2408094020669846 /loss/FRt1/detection_loss‚2.3133237302924194 /loss/FBt3/detection_loss‚3.0990281820297243 1loss/SRqqt3/detection_loss‚1.5090080985267247 0loss/SRqqt4/detection_loss‚2.368564452185775 /loss/FRt4/detection_loss‚3.0668894410133363 .loss/FBt1/detection_loss‚2.442196794918605 %loss/total_loss‚41.16766767511908 0loss/FRt1/assignment_loss‚2.3438741291562715 0loss/FRt3/assignment_loss‚3.0806687144856704 1loss/FBt1/assignment_loss‚0.33151140370111015 1loss/SRqqt2/detection_loss‚0.8559807653984298 _runtime‚7856.670760228 /loss/FRt4/assignment_loss‚3.794130573272705 0loss/FBt4/assignment_loss‚0.2103701982011374 .loss/FBt4/detection_loss‚2.323973224713252 trainer/global_step‚10049 0loss/FRt2/assignment_loss‚1.2461338593409612 1loss/SRqqt1/detection_loss‚1.7244369346234534 2loss/SRqqt1/assignment_loss‚1.6354930894794288 2loss/SRqqt2/assignment_loss‚1.2120563059613325 2loss/SRqqt4/assignment_loss‚1.6194356331587627 1loss/FBt3/assignment_loss‚0.46313649956136943 /loss/FRt3/detection_loss‚2.1272475084191873 epoch‚8 _step‚827 2loss/SRqqt3/assignment_loss‚1.2713911119582397 -loss/FRt2/detection_loss‚1.37122929124878 /loss/FBt2/detection_loss‚0.5167783314054184 % _timestamp‚1.7884557651164653e+09»9Ãßgž¿%‚(:“ ØÖæÔµ®ÖÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.435' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.287000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐM'NÀ%‚(:à ØÖæÔÛÙ«Ø disk.rbd0.in‚0! disk.rbd2.out‚268.57080078125( disk.nvme0c0n1.out‚34710.84033203125 disk.sda2.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd1.out‚ 3.986328125$ memory_percent‚94.19779777526855 cpu‚25.369425702378457 proc.cpu.threads‚17) disk./.usagePercent‚74.51077216214215# disk.dm-0.out‚34710.84033203125 disk.sda.out‚ 308.73828125 disk.sda1.in‚0 disk.nvme0n1.in‚ 199.60546875& disk.nvme0n1.out‚34710.84033203125 network.recv‚ 2.863498e+06 disk.dm-1.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd1777.in‚0) proc.memory.percent‚17.55349636077881% disk./.usageGB‚1110.2643508911133 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0' proc.memory.availableMB‚ 950.6328125 disk.dm-1.out‚ 308.73828125 disk.sda3.out‚ 308.73828125 disk.dm-0.in‚ 199.60546875 disk.rbd3.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 199.60546875# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.4455959e+07pX¢o“Á%‚(:ˆ çÖæÔ®ìùÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.368' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WéšGÂ%‚(:¼ çÖæÔÝŒÔØ' proc.memory.availableMB‚ 950.9921875% disk./.usageGB‚1110.2643585205078 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0 disk.dm-0.in‚ 199.60546875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 network.recv‚ 2.869813e+06 disk.sda1.out‚0 disk.drbd1777.in‚0 cpu‚25.368817120848806 proc.cpu.threads‚17 disk.sda.out‚ 309.1953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 199.60546875$ memory_percent‚94.19560432434082# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.456372e+07 disk.rbd3.out‚0! disk.rbd2.out‚268.57080078125" disk.nvme0c0n1.in‚ 199.60546875' disk.nvme0c0n1.out‚34809.1201171875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚34809.1201171875 disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.51077267415721 disk.sda3.out‚ 309.1953125" disk.dm-0.out‚34809.1201171875 disk.sda2.in‚0 disk.dm-1.out‚ 309.1953125¼ùäžÃ%‚(:“ öÖæÔ®™ÅÕ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚163.008' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.601600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0× ??Ä%‚(:´ öÖæÔÇÅ‹Ù$ disk./.usageGB‚1110.264389038086$ disk.nvme0n1.out‚34906.619140625! disk.dm-0.out‚34906.619140625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 309.78125 disk.rbd2.in‚0 disk.sda.in‚ 0.01171875$ memory_percent‚94.19615268707275 disk.nvme0n1.in‚ 199.60546875 disk.dm-1.in‚ 0.01171875 cpu‚25.368241741786857 network.sent‚ 2.4570141e+07) disk./.usagePercent‚74.51077472221749# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881! disk.rbd2.out‚270.31298828125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 309.78125 disk.rbd1.out‚ 3.986328125& disk.nvme0c0n1.out‚34906.619140625 network.recv‚ 2.874458e+06 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 199.60546875 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.60546875 disk.sda.out‚ 309.78125( proc.memory.availableMB‚ 950.90234375ɱ§2Å%- _wandbruntime‚7891  _runtime‚7891C•åÑÉÆ%à _step‚828 ' lr-AdamW/pg1‚1.4809510373498358e-05 ' lr-AdamW/pg2‚1.4809510373498358e-05 trainer/global_step‚10099 $ _timestamp‚1.788455799797647e+09 _runtime‚7891.352036508Îyß»ÎÇ%È _runtime‚7891.352036508 _step‚828 ' lr-AdamW/pg1‚1.4809510373498358e-05 ' lr-AdamW/pg2‚1.4809510373498358e-05 trainer/global_step‚10099 $ _timestamp‚1.788455799797647e+09¼KEQÙ2È%- _wandbruntime‚7891  _runtime‚7891 AØ=ç'É%áO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134105273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚829 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337788082 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚572335473865728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8877213721600 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 7891.35300903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697983488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48324305 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134104821 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572335456826368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041620992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337788082 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚572335473865728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190752622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8877213721600 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 7891.35300903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8463697983488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48324305 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134104821 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚572335456826368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8463041620992 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338636 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878233815552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337800512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596265809829376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190791140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135450624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572390377996288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/segment7D©ú.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 7892.02078879 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8463861048832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8432698904384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791140 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650722172406272 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134117597 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10099 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190771076 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517424640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878070204928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788455800465952e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025'Ø» õ&Í%ïM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8878070204928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788455800465952e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48329377 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190771076 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596265809829376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338636 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878233815552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337800512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596265809829376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190791140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596265135450624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚572390377996288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 7892.02078879 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8463861048832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8432698904384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190791140 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650722172406272 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134117597 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚10099 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190771076 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8464517424640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521¾Ó·Ãz2Î%- _wandbruntime‚7892  _runtime‚7892Zèú![Ï%Õ 0loss/FRt2/assignment_loss‚1.1477837288379669 2loss/SRqqt2/assignment_loss‚0.9996816987956041 2loss/SRqqt3/assignment_loss‚1.5162537884540284 2loss/SRqqt4/assignment_loss‚2.0845145732164383 /loss/FRt1/detection_loss‚2.8981225700939404 2loss/SRqqt1/assignment_loss‚1.3581822276860476 1loss/SRqqt2/detection_loss‚1.1672683079776012 .loss/FBt1/detection_loss‚3.268673337996006 0loss/FBt1/assignment_loss‚0.3627238663029857 1loss/FBt2/assignment_loss‚0.13638376974551053 0loss/FBt3/assignment_loss‚0.2724110775698836 1loss/SRqqt3/detection_loss‚1.8145950986788824 $ _timestamp‚1.788455800467932e+09 .loss/FBt4/detection_loss‚2.986671324287142 0loss/FRt1/assignment_loss‚3.1370617151260376 1loss/FBt4/assignment_loss‚0.13418069225735962 .loss/FRt4/detection_loss‚3.085648951707063 0loss/SRqqt1/detection_loss‚1.757609002441168 .loss/FBt3/detection_loss‚2.273648275778844 &loss/total_loss‚44.074846222795856 _runtime‚7892.022244773 /loss/FRt2/detection_loss‚1.8898749624192714 -loss/FRt3/detection_loss‚2.20171215547169 0loss/FBt2/detection_loss‚0.39334490416842705 epoch‚8 trainer/global_step‚10099 _step‚831 0loss/FRt3/assignment_loss‚3.6608284880285678 /loss/FRt4/assignment_loss‚3.114379401560183 1loss/SRqqt4/detection_loss‚2.4132923041952066C Q#`Ð%Ú 0loss/FBt1/assignment_loss‚0.3627238663029857 1loss/FBt2/assignment_loss‚0.13638376974551053 0loss/FBt3/assignment_loss‚0.2724110775698836 1loss/SRqqt3/detection_loss‚1.8145950986788824 $ _timestamp‚1.788455800467932e+09 .loss/FBt4/detection_loss‚2.986671324287142 0loss/FRt1/assignment_loss‚3.1370617151260376 1loss/FBt4/assignment_loss‚0.13418069225735962 .loss/FRt4/detection_loss‚3.085648951707063 0loss/SRqqt1/detection_loss‚1.757609002441168 .loss/FBt3/detection_loss‚2.273648275778844 &loss/total_loss‚44.074846222795856 _runtime‚7892.022244773 /loss/FRt2/detection_loss‚1.8898749624192714 -loss/FRt3/detection_loss‚2.20171215547169 0loss/FBt2/detection_loss‚0.39334490416842705 epoch‚8 trainer/global_step‚10099 _step‚831 0loss/FRt3/assignment_loss‚3.6608284880285678 /loss/FRt4/assignment_loss‚3.114379401560183 1loss/SRqqt4/detection_loss‚2.4132923041952066 0loss/FRt2/assignment_loss‚1.1477837288379669 2loss/SRqqt2/assignment_loss‚0.9996816987956041 2loss/SRqqt3/assignment_loss‚1.5162537884540284 2loss/SRqqt4/assignment_loss‚2.0845145732164383 /loss/FRt1/detection_loss‚2.8981225700939404 2loss/SRqqt1/assignment_loss‚1.3581822276860476 1loss/SRqqt2/detection_loss‚1.1672683079776012 .loss/FBt1/detection_loss‚3.268673337996006¿®G—žÑ%‚(:“ …×æÔ­áìÖ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.711' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.342200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ûñeJÒ%‚(:¿ …×æÔ«ˆ¬Ø disk.rbd1.out‚ 3.986328125) disk./.usagePercent‚74.51077472221749 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 310.23046875' disk.nvme0c0n1.out‚34907.2412109375% disk.nvme0n1.out‚34907.2412109375 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.out‚ 310.23046875# proc.memory.rssMB‚ 2875.96484375$ disk./.usageGB‚1110.264389038086 disk.rbd0.out‚0$ memory_percent‚94.19610500335693) proc.memory.percent‚17.55349636077881 cpu‚25.36758166252986 disk.nvme0n1.in‚ 199.60546875 disk.dm-1.out‚ 310.23046875 disk.rbd1.in‚0 network.sent‚ 2.4575711e+07 network.recv‚ 2.878955e+06" disk.nvme0c0n1.in‚ 199.60546875 disk.dm-0.in‚ 199.60546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875( proc.memory.availableMB‚ 950.91015625 disk.sda1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.rbd2.out‚271.88720703125 disk.drbd1777.in‚0 disk.rbd2.in‚0" disk.dm-0.out‚34907.2412109375}n»Þ“Ó%‚(:ˆ ”×æÔˆÛÕ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚105.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðŠ*DÔ%‚(:¹ ”×æÔï ¼Ø disk.rbd1.out‚ 3.986328125 disk.sda.out‚ 310.984375 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚35004.6123046875 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881 disk.dm-0.in‚ 199.60546875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.01171875( proc.memory.availableMB‚ 950.94921875 proc.cpu.threads‚17 disk.sda1.out‚0! disk.rbd2.out‚272.02783203125$ memory_percent‚94.19586658477783 network.recv‚ 2.886324e+06 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 199.60546875) disk./.usagePercent‚74.51077472221749 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.out‚ 310.984375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚35004.6123046875 network.sent‚ 2.4684672e+07" disk.nvme0c0n1.in‚ 199.60546875 disk.dm-1.out‚ 310.984375# proc.memory.rssMB‚ 2875.96484375 cpu‚25.36698128846985" disk.dm-0.out‚35004.6123046875 disk.sda2.in‚0$ disk./.usageGB‚1110.264389038086ý ù2Õ%- _wandbruntime‚7927  _runtime‚7927¸ÁÜÊÖ%Ä % _timestamp‚1.7884558355726984e+09 _runtime‚7927.127078921 _step‚832 ' lr-AdamW/pg1‚1.4094367449844677e-05 ' lr-AdamW/pg2‚1.4094367449844677e-05 trainer/global_step‚10149â™÷qÏ×%É _runtime‚7927.127078921 _step‚832 ' lr-AdamW/pg1‚1.4094367449844677e-05 ' lr-AdamW/pg2‚1.4094367449844677e-05 trainer/global_step‚10149 % _timestamp‚1.7884558355726984e+09Àý\‡2Ø%- _wandbruntime‚7927  _runtime‚7927Ä^ñ³è'Ù%âO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785284376576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8504683789312 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504027426816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191698712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275097600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147716220 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678674 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575080680393728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134742082 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8920102174208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚583553381163196 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884558355734448e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48553633 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147716222 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027426816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599067275097600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674‚»õÚ%çO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147716220 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191698712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275097600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590562591308288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147716220 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678674 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575080680393728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134742082 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8920102174208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚583553381163196 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884558355734448e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48553633 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147716222 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027426816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599067275097600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785284376576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8504683789312 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504027426816 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134754881 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚10149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921123990528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644926639084544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846854656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147730236 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8473534243792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚834 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48558262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147730236 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8920958282752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590618944347136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚583609101728720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847597367296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503230464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788455836249211e+09¢Ùö&Ý%ðM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8920958282752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590618944347136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚583609101728720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191697128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653847597367296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8505503230464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788455836249211e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773198848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206044 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134755348 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191697128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134754881 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚10149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921123990528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644926639084544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8504846854656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147730236 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8473534243792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚834 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86197086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48558262 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147730236—n2Þ%- _wandbruntime‚7927  _runtime‚7927ÄpYß%Ó 1loss/SRqqt3/assignment_loss‚1.492508347785356 1loss/FBt2/assignment_loss‚0.16102033967850835 % _timestamp‚1.7884558362509508e+09 0loss/FRt2/assignment_loss‚1.0338863001929388 0loss/FRt3/assignment_loss‚2.8936721881230674 _step‚835 &loss/total_loss‚41.354000513209684 1loss/FBt4/assignment_loss‚0.09951210477655488 /loss/FRt4/assignment_loss‚3.306979659725638 0loss/FBt3/assignment_loss‚0.7517286164345153 /loss/FBt1/detection_loss‚2.2788891962596347 1loss/SRqqt1/detection_loss‚1.5599511941445285 1loss/SRqqt3/detection_loss‚1.7756211414348846 /loss/FBt2/detection_loss‚0.6554984823394155 trainer/global_step‚10149 _runtime‚7927.805247793 0loss/FRt1/assignment_loss‚2.8768715833624205 2loss/SRqqt2/assignment_loss‚1.0022497338617944 0loss/FBt1/assignment_loss‚0.4128313284766461 /loss/FRt1/detection_loss‚2.3998073283582926 /loss/FRt4/detection_loss‚2.9092355440644657 0loss/SRqqt2/detection_loss‚1.012107551511791 1loss/SRqqt4/detection_loss‚2.3401918578892946 .loss/FBt3/detection_loss‚2.502806152616228 /loss/FBt4/detection_loss‚3.1372382640838623 epoch‚8 /loss/FRt2/detection_loss‚1.4904530970034775 1loss/SRqqt1/assignment_loss‚1.301308896167782 /loss/SRqqt4/assignment_loss‚1.9909832556732 -loss/FRt3/detection_loss‚1.96864834924539;l±‰^à%Ø /loss/FRt4/detection_loss‚2.9092355440644657 0loss/SRqqt2/detection_loss‚1.012107551511791 1loss/SRqqt4/detection_loss‚2.3401918578892946 .loss/FBt3/detection_loss‚2.502806152616228 /loss/FBt4/detection_loss‚3.1372382640838623 epoch‚8 /loss/FRt2/detection_loss‚1.4904530970034775 1loss/SRqqt1/assignment_loss‚1.301308896167782 /loss/SRqqt4/assignment_loss‚1.9909832556732 -loss/FRt3/detection_loss‚1.96864834924539 1loss/SRqqt3/assignment_loss‚1.492508347785356 1loss/FBt2/assignment_loss‚0.16102033967850835 % _timestamp‚1.7884558362509508e+09 0loss/FRt2/assignment_loss‚1.0338863001929388 0loss/FRt3/assignment_loss‚2.8936721881230674 _step‚835 &loss/total_loss‚41.354000513209684 1loss/FBt4/assignment_loss‚0.09951210477655488 /loss/FRt4/assignment_loss‚3.306979659725638 0loss/FBt3/assignment_loss‚0.7517286164345153 /loss/FBt1/detection_loss‚2.2788891962596347 1loss/SRqqt1/detection_loss‚1.5599511941445285 1loss/SRqqt3/detection_loss‚1.7756211414348846 /loss/FBt2/detection_loss‚0.6554984823394155 trainer/global_step‚10149 _runtime‚7927.805247793 0loss/FRt1/assignment_loss‚2.8768715833624205 2loss/SRqqt2/assignment_loss‚1.0022497338617944 0loss/FBt1/assignment_loss‚0.4128313284766461 /loss/FRt1/detection_loss‚2.3998073283582926à †mžá%‚(:“ £×æÔÍÜÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.633' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.526600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 X~E)â%‚(: £×æԯăÙ% disk./.usageGB‚1110.2646560668945 disk.sda1.out‚0 disk.dm-1.out‚ 311.49609375 disk.rbd1.in‚0# memory_percent‚94.1988468170166# disk.dm-0.out‚35005.06298828125 disk.sda2.out‚0# proc.memory.rssMB‚ 2875.96484375( disk.nvme0c0n1.out‚350|G´$05.06298828125' proc.memory.availableMB‚ 950.4609375 network.sent‚ 2.4689909e+07 disk.sda3.out‚ 311.49609375 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚273.62548828125& disk.nvme0n1.out‚35005.06298828125 disk.sda.out‚ 311.49609375) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.51079264274489 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 199.60546875 disk.dm-1.in‚ 0.01171875 network.recv‚ 2.889843e+06 disk.sda.in‚ 0.01171875 cpu‚25.366380387757218 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 199.60546875 disk.drbd1777.in‚0 disk.dm-0.in‚ 199.60546875 disk.sda2.in‚0 disk.rbd1.out‚ 3.986328125 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17‘?´Džã%‚(:“ ²×æÔ¡µ²Õ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.179199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ$˜=Dä%‚(:¹ ²×æÔã€ÙØ disk.sda1.out‚0$ memory_percent‚94.19698715209961 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚35102.56640625 disk.dm-0.in‚ 199.60546875 disk.sda2.out‚0 disk.rbd1.out‚ 3.986328125 network.recv‚ 2.896814e+06% disk.nvme0c0n1.out‚35102.56640625 disk.nvme0n1.in‚ 199.60546875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17) disk./.usagePercent‚74.51079136270722% disk./.usageGB‚1110.2646369934082 disk.drbd1777.out‚0 disk.sda2.in‚0 network.sent‚ 2.4798603e+07 disk.dm-1.in‚ 0.01171875 disk.sda3.out‚ 311.81640625) proc.memory.percent‚17.55349636077881 cpu‚25.365787577538924 disk.dm-1.out‚ 311.81640625 disk.sda.out‚ 311.81640625 disk.rbd0.out‚0& proc.memory.availableMB‚ 950.765625" disk.nvme0c0n1.in‚ 199.60546875 disk.dm-0.out‚35102.56640625 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0! disk.rbd2.out‚273.70361328125 disk.rbd3.out‚0ŽÔ¦2å%- _wandbruntime‚7962  _runtime‚7962ìm ûÉæ%à _runtime‚7962.602502994 _step‚836 ' lr-AdamW/pg1‚1.3395489574607616e-05 ' lr-AdamW/pg2‚1.3395489574607616e-05 trainer/global_step‚10199 $ _timestamp‚1.788455871048115e+09ÛFÅqÎç%È ' lr-AdamW/pg1‚1.3395489574607616e-05 ' lr-AdamW/pg2‚1.3395489574607616e-05 trainer/global_step‚10199 $ _timestamp‚1.788455871048115e+09 _runtime‚7962.602502994 _step‚836Ä–z˜Ø2è%- _wandbruntime‚7962  _runtime‚7962©$ Éç'é%áO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148417020 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48782370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912845824 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135379238 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚7962.603444077 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656910703681024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148236 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8545013232640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 $ _timestamp‚1.788455871048855e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8962992984064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135379688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48782370 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912845824 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135379238 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚7962.603444077 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656910703681024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8513553148236 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8545013232640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 $ _timestamp‚1.788455871048855e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8962992984064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135379688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148431038 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410947072 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8515024021164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8546489036288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚838 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135391906 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8514369583200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚593436596289536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚593436578286592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8964012766720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489036288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648009175967744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8545832660480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192623180 _runtime‚7963.285251526 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973212674560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643244 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48786863 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 [MDeviceStatsMonitor.on_train_batch_end/ina®Š-Ective_split_bytes.small_pool.current‚ 169902080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085325824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430'SÉ÷&í%ñM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148431038 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚601982410947072 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8515024021164 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8546489036288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚838 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586395815073452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135391906 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8514369583200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86605484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚593436596289536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚593436578286592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8964012766720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148431038 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8546489036288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648009175967744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192623180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8545832660480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192623180 _runtime‚7963.285251526 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚656973212674560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586395143596128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643244 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48786863 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085325824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚593436578286592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8545832660480 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999038464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660036094364672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651030086053376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884559068094559e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655400960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136017124 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036271442944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚589126136375128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚7998.364049407 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521Ù2­Õü%çO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596197895193088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193550816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651030086053376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651030062113280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884559068094559e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655400960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136017124 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036271442944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚589126136375128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚7998.364049407 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 _step‚841 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487644 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49010978 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589125464898012 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8585999038464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660036094364672Éfî¡ñ2ý%- _wandbruntime‚7999  _runtime‚7999€ƒ¸Yò&þ%ìM _runtime‚7999.039859732 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10249 % _timestamp‚1.7884559074848368e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149131838 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580625997580288 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136029708 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007045809152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660098782600192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660098584563712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49015822 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193569296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596254230228992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136029252 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944ÇÖÍB÷&ÿ%ñM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149131838 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580625997580288 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136029708 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007045809152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660098782600192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660098584563712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701134 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49015822 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193569296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87013886 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596254230228992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136029252 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚7999.039859732 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10249 % _timestamp‚1.7884559074848368e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928Êq6â´2€&- _wandbruntime‚7999  _runtime‚7999_ðèX&Ò 2loss/SRqqt4/assignment_loss‚2.1564163230359554 0loss/FBt2/assignment_loss‚0.0807911726245582 /loss/FRt3/detection_loss‚2.4406189226441914 /loss/FBt2/detection_loss‚0.7303539717922499 %loss/total_loss‚42.32253421751717 trainer/global_step‚10249 % _timestamp‚1.7884559074870212e+09 _step‚843 epoch‚8 0loss/FRt2/assignment_loss‚1.1656093063025639 /loss/FRt4/assignment_loss‚3.624790218141344 1loss/SRqqt2/assignment_loss‚1.051140435039997 .loss/FRt4/detection_loss‚3.019509591438152 1loss/SRqqt2/detection_loss‚0.9048009225493902 /loss/FBt1/detection_loss‚2.4888866345087686 _runtime‚7999.041340402 /loss/FRt1/assignment_loss‚2.598444564001901 /loss/FRt3/assignment_loss‚3.255866086483002 1loss/SRqqt1/assignment_loss‚1.268221411488152 1loss/FBt3/assignment_loss‚0.34215973204533967 /loss/FBt3/detection_loss‚3.0515903575079784 /loss/FBt4/detection_loss‚2.7128447393576303 0loss/FBt4/assignment_loss‚0.2971618199565758 .loss/FRt1/detection_loss‚2.219406099039681 1loss/SRqqt1/detection_loss‚1.5620502002005066 2loss/SRqqt3/assignment_loss‚1.5048291617814376 0loss/SRqqt3/detection_loss‚1.588176117715787 0loss/SRqqt4/detection_loss‚2.547452537076814 1loss/FBt1/assignment_loss‚0.18365606379423602 .loss/FRt2/detection_loss‚1.527757828990961”X]‚&× 1loss/FBt3/assignment_loss‚0.34215973204533967 /loss/FBt3/detection_loss‚3.0515903575079784 /loss/FBt4/detection_loss‚2.7128447393576303 0loss/FBt4/assignment_loss‚0.2971618199565758 .loss/FRt1/detection_loss‚2.219406099039681 1loss/SRqqt1/detection_loss‚1.5620502002005066 2loss/SRqqt3/assignment_loss‚1.5048291617814376 0loss/SRqqt3/detection_loss‚1.588176117715787 0loss/SRqqt4/detection_loss‚2.547452537076814 1loss/FBt1/assignment_loss‚0.18365606379423602 .loss/FRt2/detection_loss‚1.527757828990961 2loss/SRqqt4/assignment_loss‚2.1564163230359554 0loss/FBt2/assignment_loss‚0.0807911726245582 /loss/FRt3/detection_loss‚2.4406189226441914 /loss/FBt2/detection_loss‚0.7303539717922499 %loss/total_loss‚42.32253421751717 trainer/global_step‚10249 % _timestamp‚1.7884559074870212e+09 _step‚843 epoch‚8 0loss/FRt2/assignment_loss‚1.1656093063025639 /loss/FRt4/assignment_loss‚3.624790218141344 1loss/SRqqt2/assignment_loss‚1.051140435039997 .loss/FRt4/detection_loss‚3.019509591438152 1loss/SRqqt2/detection_loss‚0.9048009225493902 /loss/FBt1/detection_loss‚2.4888866345087686 _runtime‚7999.041340402 /loss/FRt1/assignment_loss‚2.598444564001901 /loss/FRt3/assignment_loss‚3.255866086483002 1loss/SRqqt1/assignment_loss‚1.268221411488152ËlГžƒ&‚(:“ î×æÔƒäÙÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚153.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.665800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N)lÏ>„&‚(:³ î×æÔ®†„Ù disk.drbd1777.out‚0! disk.dm-0.out‚35396.638671875 network.recv‚ 2.916298e+06 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda.in‚ 0.01171875 network.sent‚ 2.4923656e+07) disk./.usagePercent‚74.51078880263188 disk.rbd0.in‚0 proc.memory.availableMB‚951 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 313.86328125 disk.dm-0.in‚ 199.60546875 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 313.86328125 disk.rbd1.out‚ 3.986328125 disk.sda3.out‚ 313.86328125 disk.rbd2.in‚0! disk.rbd2.out‚278.69580078125 cpu‚25.363427452630127 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875" disk.nvme0c0n1.in‚ 199.60546875& disk.nvme0c0n1.out‚35396.638671875 disk.nvme0n1.in‚ 199.60546875% disk./.usageGB‚1110.2645988464355 disk.rbd3.out‚0# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 disk.rbd0.out‚0$ disk.nvme0n1.out‚35396.638671875" memory_percent‚94.195556640625 proc.cpu.threads‚17!-Ú/ž…&‚(:“ ý×æÔ›ÏÕ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.520600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sY'M†&‚(: ý×æÔÏá£Ø network.recv‚ 2.922893e+06 disk.sda.out‚ 314.42578125 disk.sda3.in‚ 0.01171875 disk.sda2.out‚0# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.5031707e+07! disk.rbd2.out‚278.72314453125& disk.nvme0n1.out‚35397.84814453125 disk.drbd1777.in‚0( proc.memory.availableMB‚ 951.25390625 disk.dm-0.in‚ 199.60546875 disk.dm-1.in‚ 0.01171875$ memory_percent‚94.19400691986084 cpu‚25.362848346294737 disk.rbd3.in‚0( disk.nvme0c0n1.out‚35397.84814453125# disk.dm-0.out‚35397.84814453125 disk.dm-1.out‚ 314.42578125 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.51078931464694# disk./.usageGB‚1110.26460647583 disk.rbd0.in‚0 disk.sda3.out‚ 314.42578125 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881 disk.rbd1.out‚ 3.986328125 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 199.60546875 disk.nvme0n1.in‚ 199.60546875ºìH­2‡&- _wandbruntime‚8033  _runtime‚8033ëL+pʈ&Ä ' lr-AdamW/pg1‚1.2047108129826328e-05 ' lr-AdamW/pg2‚1.2047108129826328e-05 trainer/global_step‚10299 % _timestamp‚1.7884559422792847e+09 _runtime‚8033.833667776 _step‚844qÜ}ºÏ‰&É % _timestamp‚1.7884559422792847e+09 _runtime‚8033.833667776 _step‚844 ' lr-AdamW/pg1‚1.2047108129826328e-05 ' lr-AdamW/pg2‚1.2047108129826328e-05 trainer/global_step‚10299Ì8Ó2Š&- _wandbruntime‚8033  _runtime‚8033ÕIÛ}²‹&âO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1}[©«6'63111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8627641206784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9049057494528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚583316282938368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49238322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149818620 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599015529132544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8627641206784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49237876 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚845 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984844288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136651992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8595223827052 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663161656491008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188342272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595878264808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513976832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149818620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583316299977728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476868 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654112598996480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607643188342272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765420 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984844288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136651992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8595223827052 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663161656491008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188342272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8595878264808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10299 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513976832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149818620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583316299977728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194476868 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654112598996480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607643188342272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654112622936576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591911506765420 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚583371204108288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599071900174336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591967227330944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460647936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583371187068928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136664715 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 # _timestamp‚1.78845594294552e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663224172275712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050081957888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804272128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149832638 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460647936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 trainer/global_step‚10299 _runtime‚8034.500359705 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599071882171392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596694699980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚846 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654174273674240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194495348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136665169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9049922541568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149832636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686443520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149832638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663224355632128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¶{1˜ &ïM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607700360822272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8596040262016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚583371204108288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599071900174336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591967227330944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8628460647936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583371187068928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_΃Õp]pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136664715 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 # _timestamp‚1.78845594294552e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663224172275712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050081957888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804272128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149832638 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460647936 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 trainer/global_step‚10299 _runtime‚8034.500359705 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599071882171392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596694699980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚846 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654174273674240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194495348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136665169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9049922541568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149832636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686443520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149832638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663224355632128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49242883 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8667970650112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 % _timestamp‚1.7884559780504138e+09 _runtime‚8069.605165072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚594698220109944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49467657 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610501826090496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195402920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49467201 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092402766336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151725056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137289717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657195159819776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ü,X¨À!œ&çO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚601833181074944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345902302 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627012608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092232850944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195382882 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636713604216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137290175 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195402920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150519420 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8667970650112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 % _timestamp‚1.7884559780504138e+09 _runtime‚8069.605165072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚594698220109944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49467657 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610501826090496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195402920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49467201 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092402766336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151725056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137289717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657195159819776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666287562586112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586061506505728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944ÑA»ò¦“&‚(:ˆ ªØæÔ€¢æÕ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôš,è?ž&‚(:´ ªØæÔü—‰Ù disk.dm-1.in‚ 0.01171875) proc.memory.percent‚17.55349636077881 cpu‚25.361104281605435) disk./.usagePercent‚74.51082592372433 disk.sda.out‚ 315.9921875 disk.dm-0.in‚ 199.60546875 disk.dm-0.out‚ 35593.2265625# proc.memory.rssMB‚ 2875.96484375 network.recv‚ 2.938565e+06$ memory_percent‚94.19729709625244$ disk./.usageGB‚1110.265151977539! disk.rbd2.out‚282.05908203125" disk.nvme0n1.out‚ 35593.2265625 disk.sda2.in‚0( proc.memory.availableMB‚ 950.71484375 network.sent‚ 2.5151984e+07 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 315.9921875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 199.60546875 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 199.60546875 disk.dm-1.out‚ 315.9921875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.sda.in‚ 0.01171875 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 35593.2265625 disk.drbd1777.out‚0€%Ô2Ÿ&- _wandbruntime‚8070  _runtime‚8070äÚ”ò& &ìM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137302427 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚8070.280562312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093103303680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093260622848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49471743 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666349889921024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669446453760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324191744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675676 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Îf÷&¡&ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚8070.280562312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093103303680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093260622848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49471743 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666349889921024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669446453760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚610558324191744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657256786617344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚594753940675676 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657256810557440 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137302427 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928ÒRØØ2¢&- _wandbruntime‚8070  _runtime‚8070—ÇkVZ£&Ô /loss/FRt1/assignment_loss‚2.845734923415714 0loss/FRt4/assignment_loss‚3.9728910218585622 1loss/SRqqt1/assignment_loss‚1.147515515756348 %loss/total_loss‚42.97748785010264 trainer/global_step‚10349 % _timestamp‚1.7884559787273426e+09 1loss/SRqqt4/assignment_loss‚2.211883114083321 1loss/SRqqt2/detection_loss‚1.1800156904215162 0loss/SRqqt3/detection_loss‚1.956947846987913 1loss/SRqqt4/detection_loss‚3.1396657153964043 .loss/FBt3/detection_loss‚2.680211056023836 0loss/FRt2/assignment_loss‚1.1052652829223208 .loss/FRt2/detection_loss‚1.577798811580848 .loss/FBt2/detection_loss‚0.745209609302751 1loss/FBt4/assignment_loss‚0.12702566134975868 1loss/SRqqt1/detection_loss‚1.6503969411487165 epoch‚8 1loss/FBt1/assignment_loss‚0.07653646841935806 1loss/FBt2/assignment_loss‚0.08599289422132188 /loss/FRt3/detection_loss‚2.1015881287808322 .loss/FRt4/detection_loss‚3.232207493348555 .loss/FBt1/detection_loss‚2.921395778656006 1loss/SRqqt2/assignment_loss‚0.892945077591999 1loss/FBt3/assignment_loss‚0.03433866333216429 /loss/FRt1/detection_loss‚2.2395118567678662 /loss/FBt4/detection_loss‚2.7619880363345146 _runtime‚8070.281609756 _step‚851 0loss/FRt3/assignment_loss‚3.0113681238524768 2loss/SRqqt3/assignment_loss‚1.2790541385495386–¼á&_¤&Ù 0loss/SRqqt3/detection_loss‚1.956947846987913 1loss/SRqqt4/detection_loss‚3.1396657153964043 .loss/FBt3/detection_loss‚2.680211056023836 0loss/FRt2/assignment_loss‚1.1052652829223208 .loss/FRt2/detection_loss‚1.577798811580848 .loss/FBt2/detection_loss‚0.745209609302751 1loss/FBt4/assignment_loss‚0.12702566134975868 1loss/SRqqt1/detection_loss‚1.6503969411487165 epoch‚8 1loss/FBt1/assignment_loss‚0.07653646841935806 1loss/FBt2/assignment_loss‚0.08599289422132188 /loss/FRt3/detection_loss‚2.1015881287808322 .loss/FRt4/detection_loss‚3.232207493348555 .loss/FBt1/detection_loss‚2.921395778656006 1loss/SRqqt2/assignment_loss‚0.892945077591999 1loss/FBt3/assignment_loss‚0.03433866333216429 /loss/FRt1/detection_loss‚2.2395118567678662 /loss/FBt4/detection_loss‚2.7619880363345146 _runtime‚8070.281609756 _step‚851 0loss/FRt3/assignment_loss‚3.0113681238524768 2loss/SRqqt3/assignment_loss‚1.2790541385495386 /loss/FRt1/assignment_loss‚2.845734923415714 0loss/FRt4/assignment_loss‚3.9728910218585622 1loss/SRqqt1/assignment_loss‚1.147515515756348 %loss/total_loss‚42.97748785010264 trainer/global_step‚10349 % _timestamp‚1.7884559787273426e+09 1loss/SRqqt4/assignment_loss‚2.211883114083321 1loss/SRqqt2/detection_loss‚1.1800156904215162ÓKDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9135270311424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669412943074304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚660277672762880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597484261977352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49695982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789473280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196308934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_trainÓmÔ0_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151220222 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137926900 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612818432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9135442323968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8677548943624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463838720O1¥ì'®&æO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505868 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9135442323968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8677548943624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463838720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49695528 _step‚853 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9135270311424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669412943074304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚660277672762880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597484261977352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49695982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789473280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196308934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151220222 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137926900 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612818432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88239084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136303434752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49699998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137939082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136146115584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775883776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49700464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8105.948067056 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327388 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588861617164288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347561624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137939550 $ _timestamp‚1.788456014393182e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636318720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚854 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604707204059136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378796 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8710432259584IóYž±&ðM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚660339347440640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136303434752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49699998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137939082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136146115584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775883776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49700464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8105.948067056 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327388 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588861617164288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347561624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137939550 $ _timestamp‚1.788456014393182e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636318720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMoneƒ=TXitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚854 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604707204059136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378796 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597539982543084 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8710432259584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775883776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88239084 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944Ö×ðË+2²&- _wandbruntime‚8105  _runtime‚8105&s<¥V³&Ð 0loss/SRqqt1/detection_loss‚1.639054713100195 1loss/SRqqt3/detection_loss‚1.9320149272680283 1loss/SRqqt4/detection_loss‚2.9213788417669444 .loss/FBt1/detection_loss‚2.440428704023361 &loss/total_loss‚42.198635871523535 % _timestamp‚1.7884560143949091e+09 0loss/FRt1/assignment_loss‚2.5251340759106173 /loss/FRt3/assignment_loss‚2.618660220503807 0loss/SRqqt2/assignment_loss‚1.13537855927522 0loss/FBt3/assignment_loss‚0.2928518061671497 1loss/FBt4/assignment_loss‚0.17344171114382334 .loss/FRt2/detection_loss‚1.525685483727742 1loss/SRqqt1/assignment_loss‚1.510735233798623 1loss/FBt1/assignment_loss‚0.21574026507732924 1loss/FBt2/assignment_loss‚0.15109865945193002 .loss/FRt3/detection_loss‚2.233792404457927 0loss/SRqqt2/detection_loss‚1.032570281190177 _runtime‚8105.949163752 _step‚855 1loss/SRqqt3/assignment_loss‚1.444148610484727 .loss/FRt1/detection_loss‚2.354008014576557 /loss/FBt2/detection_loss‚0.6628721557170626 epoch‚8 .loss/FRt4/detection_loss‚3.147563112633569 /loss/FBt3/detection_loss‚2.8979296907782555 /loss/FBt4/detection_loss‚2.4124506562948227 trainer/global_step‚10399 2loss/SRqqt4/assignment_loss‚1.8659640696759408 0loss/FRt2/assignment_loss‚1.3058505985471938 .loss/FRt4/assignment_loss‚3.75988307595253¸Ü²³[´&Õ trainer/global_step‚10399 2loss/SRqqt4/assignment_loss‚1.8659640696759408 0loss/FRt2/assignment_loss‚1.3058505985471938 .loss/FRt4/assignment_loss‚3.75988307595253 0loss/SRqqt1/detection_loss‚1.639054713100195 1loss/SRqqt3/detection_loss‚1.9320149272680283 1loss/SRqqt4/detection_loss‚2.9213788417669444 .loss/FBt1/detection_loss‚2.440428704023361 &loss/total_loss‚42.198635871523535 % _timestamp‚1.7884560143949091e+09 0loss/FRt1/assignment_loss‚2.5251340759106173 /loss/FRt3/assignment_loss‚2.618660220503807 0loss/SRqqt2/assignment_loss‚1.13537855927522 0loss/FBt3/assignment_loss‚0.2928518061671497 1loss/FBt4/assignment_loss‚0.17344171114382334 .loss/FRt2/detection_loss‚1.525685483727742 1loss/SRqqt1/assignment_loss‚1.510735233798623 1loss/FBt1/assignment_loss‚0.21574026507732924 1loss/FBt2/assignment_loss‚0.15109865945193002 .loss/FRt3/detection_loss‚2.233792404457927 0loss/SRqqt2/detection_loss‚1.032570281190177 _runtime‚8105.949163752 _step‚855 1loss/SRqqt3/assignment_loss‚1.444148610484727 .loss/FRt1/detection_loss‚2.354008014576557 /loss/FBt2/detection_loss‚0.6628721557170626 epoch‚8 .loss/FRt4/detection_loss‚3.147563112633569 /loss/FBt3/detection_loss‚2.8979296907782555 /loss/FBt4/detection_loss‚2.4124506562948227××h“µ&‚(:ˆ ×ØæÔ›«òÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e˜ G¶&‚(:¼ ×ØæÔõŒ¼Ø' disk./.usagePercent‚74.510823363649 disk.sda3.out‚ 319.7734375 disk.rbd2.out‚285.4306640625 disk.rbd3.out‚0 network.recv‚ 2.955829e+06% disk./.usageGB‚1110.2651138305664 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚35793.60595703125 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 3.986328125 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 199.60546875 disk.sda2.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 950.7265625 cpu‚25.35935765232343 disk.sda1.in‚0& disk.nvme0n1.out‚35793.60595703125 disk.drbd1777.out‚0 disk.dm-0.in‚ 199.60546875 disk.dm-1.out‚ 319.7734375) proc.memory.percent‚17.55349636077881 disk.sda3.in‚ 0.01171875( disk.nvme0c0n1.out‚35793.60595703125 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 199.60546875 disk.sda.out‚ 319.7734375 network.sent‚ 2.5352208e+07 disk.drbd1777.in‚0$ memory_percent‚94.19722557067871£êUž·&‚(:“ æØæÔ½¼ÉÕ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.836' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.167200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J†®E¸&‚(:º æØæÔÙ‡ÝØ disk.drbd1777.out‚0 disk.dm-1.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125# proc.memory.rssMB‚ 2875.96484375 disk.dm-1.out‚ 321.3203125& disk.nvme0n1.out‚35911.22998046875( disk./.usagePercent‚74.5149020756845# disk.dm-0.out‚35911.22998046875 disk.nvme0n1.in‚ 295.5703125 network.sent‚ 2.5461054e+07 network.recv‚ 2.962543e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 321.3203125 proc.cpu.threads‚17% disk./.usageGB‚1110.3258895874023 disk.sda2.out‚0( disk.nvme0c0n1.out‚35911.22998046875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚94.1988468170166' proc.memory.availableMB‚ 950.4609375 disk.dm-0.in‚ 295.5703125 disk.rbd3.out‚0 disk.sda.out‚ 321.3203125 disk.rbd2.out‚285.4306640625! disk.nvme0c0n1.in‚ 295.5703125) proc.memory.percent‚17.55349636077881 cpu‚25.358817653223184B ÿ2¹&- _wandbruntime‚8141  _runtime‚8141ÚýÇɺ&à ' lr-AdamW/pg1‚1.0150053187102611e-05 ' lr-AdamW/pg2‚1.0150053187102611e-05 trainer/global_step‚10449 $ _timestamp‚1.788456049580038e+09 _runtime‚8141.134436916 _step‚856Þ {λ&È ' lr-AdamW/pg1‚1.0150053187102611e-05 ' lr-AdamW/pg2‚1.0150053187102611e-05 trainer/global_step‚10449 $ _timestamp‚1.788456049580038e+09 _runtime‚8141.134436916 _step‚856ØÞ 2¼&- _wandbruntime‚8141  _runtime‚8141_ˆêÈç'½&áO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598624256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49924714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49924256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663360233586176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672538770257408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151921022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672538580596224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197255024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884560495807893e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8717729845276 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607468502962688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672538770257408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151921022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8718384283032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672538580596224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚607468484959744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197255024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884560495807893e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8717729845276 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49924714 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49924256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184Ùªê’j2¿&- _wandbruntime‚8141  _runtime‚8141ÈËCv˜À&ìM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607524837998592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚663421860383744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600325352933168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚10449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418065408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138576232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601282560512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197253440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607524837998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚616276274066944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49928748 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601101301248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179398236672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179240917504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚663421884323840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197273504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49929200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454_‹÷&Á&ñM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607524837998592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚663421860383744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349188476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600325352933168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚10449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751418065408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138576232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601282560512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197253440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607524837998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚616276274066944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49928748 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601101301248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179398236672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179240917504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚663421884323840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197273504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49929200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454ÚÙi2Â&- _wandbruntime‚8141  _runtime‚8141ŸèVÃ&Ð /loss/FRt4/assignment_loss‚3.240045179020275 1loss/SRqqt2/assignment_loss‚1.001928392482006 2loss/SRqqt3/assignment_loss‚1.2348142508417368 /loss/FRt1/detection_loss‚2.1205802274511214 1loss/SRqqt1/detection_loss‚1.5816697261940975 /loss/SRqqt3/detection_loss‚1.70849156145956 % _timestamp‚1.7884560502504125e+09 _step‚859 /loss/FRt1/assignment_loss‚2.648291882048262 1loss/FBt1/assignment_loss‚0.08474269049838767 /loss/FRt4/detection_loss‚3.0704873746091668 /loss/FRt2/detection_loss‚1.6807686645045596 1loss/SRqqt4/detection_loss‚2.4118885416537523 /loss/FBt2/detection_loss‚0.5831903955322164 0loss/FRt2/assignment_loss‚1.6667322673887577 epoch‚8 1loss/SRqqt1/assignment_loss‚1.211016734475449 1loss/FBt3/assignment_loss‚0.14522941795682223 0loss/FRt3/assignment_loss‚2.9167796142247258 .loss/FBt4/detection_loss‚2.549178085543893 #loss/total_loss‚40.016347365081 trainer/global_step‚10449 _runtime‚ 8141.80471881 1loss/SRqqt2/detection_loss‚1.0355545551973884 .loss/FBt3/detection_loss‚2.984612579147021 2loss/SRqqt4/assignment_loss‚1.3272503924090415 1loss/FBt2/assignment_loss‚0.16921306147041246 0loss/FBt4/assignment_loss‚0.1069640839017335 .loss/FRt3/detection_loss‚2.164926082504039 .loss/FBt1/detection_loss‚2.371991604566574d¨ª[Ä&Õ 0loss/FRt3/assignment_loss‚2.9167796142247258 .loss/FBt4/detection_loss‚2.549178085543893 #loss/total_loss‚40.016347365081 trainer/global_step‚10449 _runtime‚ 8141.80471881 1loss/SRqqt2/detection_loss‚1.0355545551973884 .loss/FBt3/detection_loss‚2.984612579147021 2loss/SRqqt4/assignment_loss‚1.3272503924090415 1loss/FBt2/assignment_loss‚0.16921306147041246 0loss/FBt4/assignment_loss‚0.1069640839017335 .loss/FRt3/detection_loss‚2.164926082504039 .loss/FBt1/detection_loss‚2.371991604566574 /loss/FRt4/assignment_loss‚3.240045179020275 1loss/SRqqt2/assignment_loss‚1.001928392482006 2loss/SRqqt3/assignment_loss‚1.2348142508417368 /loss/FRt1/detection_loss‚2.1205802274511214 1loss/SRqqt1/detection_loss‚1.5816697261940975 /loss/SRqqt3/detection_loss‚1.70849156145956 % _timestamp‚1.7884560502504125e+09 _step‚859 /loss/FRt1/assignment_loss‚2.648291882048262 1loss/FBt1/assignment_loss‚0.08474269049838767 /loss/FRt4/detection_loss‚3.0704873746091668 /loss/FRt2/detection_loss‚1.6807686645045596 1loss/SRqqt4/detection_loss‚2.4118885416537523 /loss/FBt2/detection_loss‚0.5831903955322164 0loss/FRt2/assignment_loss‚1.6667322673887577 epoch‚8 1loss/SRqqt1/assignment_loss‚1.211016734475449 1loss/FBt3/assignment_loss‚0.14522941795682223Û“7•‘Å&‚(:† õØæÔýÝÇÕ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.n&1Æ&‚(:¦ õØæÔÙÇïØ disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1250.64453125 cpu‚25.35824986242336 network.recv‚ 2.966643e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1250.64453125 disk.rbd1.out‚ 3.986328125 disk.sda.out‚322% disk.nvme0n1.out‚35972.7392578125( proc.memory.availableMB‚ 950.42578125) proc.memory.percent‚17.55349636077881 disk.dm-1.out‚322 disk.sda.in‚ 0.01171875 disk.rbd2.out‚287.1689453125 network.sent‚ 2.5466695e+07 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1250.64453125 disk.sda2.in‚0# memory_percent‚94.1990613937378# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.out‚322' disk.nvme0c0n1.out‚35972.7392578125) disk./.usagePercent‚74.52516336967193% disk./.usageGB‚1110.4787902832031" disk.dm-0.out‚35972.7392578125 disk.dm-1.in‚ 0.01171875®÷ÑžÇ&‚(:“ „ÙæÔ›è×Õ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.826' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.565199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒnÜMÈ&‚(: „ÙæÔ­ž’Ø disk.sda1.in‚0 disk.sda3.out‚ 322.42578125 network.sent‚ 2.5575562e+07 disk.rbd1.out‚ 4.072265625 disk.rbd3.in‚0 disk.sda3.in‚ 0.01171875 disk.rbd2.in‚0% disk.nvme0n1.out‚36194.5322265625) proc.memory.percent‚17.55349636077881 disk.sda.in‚ 0.01171875 disk.sda.out‚ 322.42578125# disk.nvme0c0n1.in‚ 1452.24609375 disk.drbd1777.out‚0$ memory_percent‚94.19987201690674% disk./.usageGB‚1110.4587211608887 disk.rbd3.out‚0( proc.memory.availableMB‚ 950.29296875 disk.rbd0.out‚0 disk.rbd2.out‚287.1689453125' disk.nvme0c0n1.out‚36194.5322265625 disk.drbd1777.in‚0 disk.sda2.out‚0 network.recv‚ 2.973832e+06( disk./.usagePercent‚74.5238165140343 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875 cpu‚25.357689660503933 disk.dm-0.in‚ 1452.24609375" disk.dm-0.out‚36194.5322265625 disk.sda2.in‚0# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1452.24609375 disk.dm-1.out‚ 322.42578125(¨“É&‚(:ˆ “ÙæÔ̆ÈÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.9418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^¯?Ê&‚(:´ “ÙæÔ¶ˆØ disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.5580651e+07 disk.sda3.out‚ 323.359375 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚36296.228515625 disk.rbd1.out‚ 4.134765625 disk.nvme0n1.in‚ 1536.453125$ memory_percent‚94.19839382171631 cpu‚25.357127354024325 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0 disk.dm-0.in‚ 1536.453125! disk.dm-0.out‚36296.228515625 network.recv‚ 2.977545e+06) disk./.usagePercent‚74.52545368221575% disk./.usageGB‚1110.4831161499023 disk.rbd3.in‚0 disk.sda.out‚ 323.359375 disk.sda1.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚36296.228515625 disk.drbd1777.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0( proc.memory.availableMB‚ 950.53515625 disk.sda1.in‚0 disk.rbd2.out‚288.9111328125 disk.dm-1.out‚ 323.359375) proc.memory.percent‚17.55349636077881! disk.nvme0c0n1.in‚ 1536.453125 disk.rbd3.out‚0 disk.rbd0.in‚0ħ¯h2Ë&- _wandbruntime‚8176  _runtime‚8176Q¦ØoÈÌ& & lr-AdamW/pg1‚9.551703428519572e-06 & lr-AdamW/pg2‚9.551703428519572e-06 trainer/global_step‚10499 % _timestamp‚1.7884560854420881e+09 _runtime‚8176.996479179 _step‚860ùör6ÍÍ&Ç & lr-AdamW/pg1‚9.551703428519572e-06 & lr-AdamW/pg2‚9.551703428519572e-06 trainer/global_step‚10499 % _timestamp‚1.7884560854420881e+09 _runtime‚8176.996479179 _step‚860ÜX>·2Î&- _wandbruntime‚8176  _runtime‚8176–ìGäÏ&âO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8758565184684 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619077739335168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9221723580416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619077064969728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610286154905088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802898 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8790928067584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8176.997437074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139201056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610286136902144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139200597 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10499 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198181076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50152881 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚603056345712168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664300194304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221553665024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198181076 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚603055674235052 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728ö.¨ôí'Ð&çO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221553665024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198181076 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚603055674235052 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚594297126089728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8758565184684 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619077739335168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9221723580416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198161038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619077064969728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610286154905088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350802898 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8759219622440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8790928067584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8176.997437074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139201056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610286136902144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139200597 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10499 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198181076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50152881 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚603056345712168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚861 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664300194304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ÝEV†2Ñ&- _wandbruntime‚8177  _runtime‚8177!èjÚñ&Ò&ëM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50157910 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8791747495424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727001248256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9222580041216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403871232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139213796 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50157474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8177.672464917 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139213358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726817891840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666504421207040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222420624896 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152635836 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610342489940992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237436416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚10499 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403871232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8760036057612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198199556 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152635838 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198179492 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788456086117312e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50157910 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8791747495424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727001248256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9222580041216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152635836 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8792403871232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139213796 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8759381619648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50157474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8177.672464917 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139213358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726817891840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666504421207040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222420624896 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152635836 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610342489940992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237436416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152635838 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÞŒÜýY2Ô&- _wandbruntime‚8177  _runtime‚8177•Ù¡Ü\Õ&Ö _step‚863 2loss/SRqqt2/assignment_loss‚0.7490223380081146 0loss/FBt1/assignment_loss‚0.1576562946672008 1loss/SRqqt3/detection_loss‚1.9061956876380877 ,loss/FBt3/detection_loss‚2.5873933583498 /loss/FRt1/assignment_loss‚2.403196696577401 2loss/SRqqt4/assignment_loss‚1.6869134866985782 /loss/FRt2/detection_loss‚1.4675921157507572 &loss/total_loss‚39.397494127288006 1loss/FBt4/assignment_loss‚0.41484976311524707 .loss/FRt1/detection_loss‚2.082187346600253 1loss/SRqqt4/detection_loss‚2.3876118053650033 /loss/FBt4/detection_loss‚2.0940376445651054 trainer/global_step‚10499 _runtime‚8177.673765558 1loss/FBt2/assignment_loss‚0.07992248445634764 1loss/FBt3/assignment_loss‚0.18964263517409563 .loss/FRt3/detection_loss‚2.250272817043371 1loss/SRqqt2/detection_loss‚0.9386631197387904 .loss/FBt1/detection_loss‚2.990886221329371 0loss/FRt3/assignment_loss‚2.7928333545840065 2loss/SRqqt1/assignment_loss‚1.6115700824827444 /loss/FRt4/detection_loss‚2.9036793429404497 /loss/FBt2/detection_loss‚0.5975074919225415 0loss/FRt2/assignment_loss‚1.0206038477057118 /loss/FRt4/assignment_loss‚3.299040764570236 epoch‚8 % _timestamp‚1.7884560861194775e+09 2loss/SRqqt3/assignment_loss‚1.2690092475915498 1loss/SRqqt1/detection_loss‚1.5172061804132384—ÿµ(aÖ&Û .loss/FBt1/detection_loss‚2.990886221329371 0loss/FRt3/assignment_loss‚2.7928333545840065 2loss/SRqqt1/assignment_loss‚1.6115700824827444 /loss/FRt4/detection_loss‚2.9036793429404497 /loss/FBt2/detection_loss‚0.5975074919225415 0loss/FRt2/assignment_loss‚1.0206038477057118 /loss/FRt4/assignment_loss‚3.299040764570236 epoch‚8 % _timestamp‚1.7884560861194775e+09 2loss/SRqqt3/assignment_loss‚1.2690092475915498 1loss/SRqqt1/detection_loss‚1.5172061804132384 _step‚863 2loss/SRqqt2/assignment_loss‚0.7490223380081146 0loss/FBt1/assignment_loss‚0.1576562946672008 1loss/SRqqt3/detection_loss‚1.9061956876380877 ,loss/FBt3/detection_loss‚2.5873933583498 /loss/FRt1/assignment_loss‚2.403196696577401 2loss/SRqqt4/assignment_loss‚1.6869134866985782 /loss/FRt2/detection_loss‚1.4675921157507572 &loss/total_loss‚39.397494127288006 1loss/FBt4/assignment_loss‚0.41484976311524707 .loss/FRt1/detection_loss‚2.082187346600253 1loss/SRqqt4/detection_loss‚2.3876118053650033 /loss/FBt4/detection_loss‚2.0940376445651054 trainer/global_step‚10499 _runtime‚8177.673765558 1loss/FBt2/assignment_loss‚0.07992248445634764 1loss/FBt3/assignment_loss‚0.18964263517409563 .loss/FRt3/detection_loss‚2.250272817043371 1loss/SRqqt2/detection_loss‚0.9386631197387904ßžÞʯ‘×&‚(:† ¢ÙæÔòµåÕ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚160.36' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»µàiFØ&‚(:» ¢ÙæÔ»•¿Ø% disk./.usageGB‚1110.4850807189941 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 323.8046875 disk.sda2.out‚0 disk.sda.out‚ 323.8046875) proc.memory.percent‚17.55349636077881 cpu‚25.356563841389182 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.in‚0# disk.dm-0.out‚36320.25341796875 disk.rbd3.in‚0( proc.memory.availableMB‚ 950.30859375! disk.nvme0c0n1.in‚ 1536.453125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.134765625# proc.memory.rssMB‚ 2875.96484375 network.recv‚ 2.981879e+06 disk.drbd1777.out‚0 disk.sda2.in‚0 network.sent‚ 2.5586969e+07( disk./.usagePercent‚74.5255855260959( disk.nvme0c0n1.out‚36320.25341796875 disk.nvme0n1.in‚ 1536.453125 disk.sda.in‚ 0.01171875 disk.sda1.out‚0# memory_percent‚94.1997766494751 disk.rbd2.out‚288.9111328125& disk.nvme0n1.out‚36320.25341796875 disk.dm-0.in‚ 1536.453125 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 323.8046875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0Ý%[’Ù&‚(:‡ ±ÙæÔ–éÕ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í ª{FÚ&‚(:» ±ÙæÔÑ Ø% disk./.usageGB‚1110.4850997924805 disk.dm-0.in‚ 1536.453125 disk.sda1.out‚0 disk.sda3.out‚ 324.47265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1536.453125$ disk.nvme0n1.out‚36421.560546875 disk.drbd1777.in‚0 network.recv‚ 2.988472e+06 disk.dm-1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.52558680613356 disk.dm-1.out‚ 324.47265625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 324.47265625 disk.rbd2.in‚0 cpu‚25.355977095312852 network.sent‚ 2.5695049e+07 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚36421.560546875 proc.cpu.threads‚17 disk.sda2.out‚0$ memory_percent‚94.19949054718018! disk.dm-0.out‚36421.560546875! disk.nvme0c0n1.in‚ 1536.453125 disk.rbd1.out‚ 4.1669921875 disk.rbd2.out‚290.5869140625( proc.memory.availableMB‚ 950.35546875# proc.memory.rssMB‚ 2875.96484375 åÉ"2Û&- _wandbruntime‚8212  _runtime‚8212ß‚îÈÜ& _runtime‚8212.743205469 _step‚864 & lr-AdamW/pg1‚8.970565402726544e-06 & lr-AdamW/pg2‚8.970565402726544e-06 trainer/global_step‚10549 % _timestamp‚1.7884561211888158e+09Íüè,ÍÝ&Ç _runtime‚8212.743205469 _step‚864 & lr-AdamW/pg1‚8.970565402726544e-06 & lr-AdamW/pg2‚8.970565402726544e-06 trainer/global_step‚10549 % _timestamp‚1.7884561211888158e+09à<|bý2Þ&- _wandbruntime‚8212  _runtime‚8212–©¥è'ß&âO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790023817728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚10549 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8831913873408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50382274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702717952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377083392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8212.744276878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9264740405248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139838836 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139838390 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89456116 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621936377083392 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790209284608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8799400524092 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613103806847488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913873408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50382718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884561211895888e+09 _step‚8658åûí'à&çO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597042315578368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9264901932032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199107128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790209284608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8799400524092 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613103806847488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8831913873408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50382718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884561211895888e+09 _step‚865 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790023817728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153322620 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚10549 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8831913873408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50382274 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702717952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚621936377083392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8212.744276878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9264740405248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139838836 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139838390 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89456116 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621936377083392 J&2á&- _wandbruntime‚8213  _runtime‚8213ñZçÅâ&ìM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚8213.425591955 _step‚866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733301248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605898108145308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50387351 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153336636 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139851183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265762000384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153336636 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884561218706264e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389677056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265598389760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8832733301248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0;E¼÷&ã&ñM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚8213.425591955 _step‚866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733301248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605898108145308 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50387351 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153336636 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139851183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265762000384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153336636 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884561218706264e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389677056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265598389760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8832733301248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414âS •ù2ä&- _wandbruntime‚8213  _runtime‚8213AG ¤`å&Ú trainer/global_step‚10549 1loss/FBt4/assignment_loss‚0.27040116135030984 0loss/FRt1/assignment_loss‚2.5613577031507724 0loss/FRt3/assignment_loss‚3.3171249559292426 2loss/SRqqt3/assignment_loss‚1.7279121725085385 $loss/total_loss‚43.2470387131697 _step‚867 1loss/FBt3/assignment_loss‚0.22462827562715879 1loss/SRqqt2/detection_loss‚1.0818704565587853 epoch‚8 % _timestamp‚1.7884561218724294e+09 2loss/SRqqt2/assignment_loss‚0.9633168687495137 2loss/SRqqt4/assignment_loss‚1.7588332796096802 3loss/FBt1/assignment_loss‚0.0008117400769454738 /loss/FRt1/detection_loss‚2.3145119518768498 0loss/FRt4/assignment_loss‚3.4441695829917647 2loss/SRqqt1/assignment_loss‚1.6161294332441856 /loss/FRt3/detection_loss‚2.2223575493464103 /loss/SRqqt1/detection_loss‚1.63836790185492 /loss/FBt2/detection_loss‚0.5949835286467836 /loss/FBt4/detection_loss‚3.2910895347595215 1loss/FBt2/assignment_loss‚0.18976412707586526 1loss/SRqqt3/detection_loss‚1.6848110900870685 .loss/FBt3/detection_loss‚2.653102000554403 0loss/FRt2/assignment_loss‚1.3876002010177164 _runtime‚8213.426719429 /loss/FRt2/detection_loss‚1.6503825169787103 .loss/FRt4/detection_loss‚2.801360799320813 0loss/SRqqt4/detection_loss‚2.466809154748917 /loss/FBt1/detection_loss‚3.3853427271048226Eõyùeæ&ß 2loss/SRqqt1/assignment_loss‚1.6161294332441856 /loss/FRt3/detection_loss‚2.2223575493464103 /loss/SRqqt1/detection_loss‚1.63836790185492 /loss/FBt2/detection_loss‚0.5949835286467836 /loss/FBt4/detection_loss‚3.2910895347595215 1loss/FBt2/assignment_loss‚0.18976412707586526 1loss/SRqqt3/detection_loss‚1.6848110900870685 .loss/FBt3/detection_loss‚2.653102000554403 0loss/FRt2/assignment_loss‚1.3876002010177164 _runtime‚8213.426719429 /loss/FRt2/detection_loss‚1.6503825169787103 .loss/FRt4/detection_loss‚2.801360799320813 0loss/SRqqt4/detection_loss‚2.466809154748917 /loss/FBt1/detection_loss‚3.3853427271048226 trainer/global_step‚10549 1loss/FBt4/assignment_loss‚0.27040116135030984 0loss/FRt1/assignment_loss‚2.5613577031507724 0loss/FRt3/assignment_loss‚3.3171249559292426 2loss/SRqqt3/assignment_loss‚1.7279121725085385 $loss/total_loss‚43.2470387131697 _step‚867 1loss/FBt3/assignment_loss‚0.22462827562715879 1loss/SRqqt2/detection_loss‚1.0818704565587853 epoch‚8 % _timestamp‚1.7884561218724294e+09 2loss/SRqqt2/assignment_loss‚0.9633168687495137 2loss/SRqqt4/assignment_loss‚1.7588332796096802 3loss/FBt1/assignment_loss‚0.0008117400769454738 /loss/FRt1/detection_loss‚2.3145119518768498 0loss/FRt4/assignment_loss‚3.4441695829917647ãn‘H,“ç&‚(:ˆ ÀÙæÔ°ääÕ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©”ÅCè&‚(:¸ ÀÙæÔѬÙ network.recv‚ 2.993304e+06 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0 disk.sda.out‚ 324.8359375 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 1536.453125 disk.nvme0n1.in‚ 1536.453125 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚36519.435546875 disk.dm-0.in‚ 1536.453125! disk.dm-0.out‚36519.435546875$ memory_percent‚94.20197010040283 network.sent‚ 2.5701386e+07 disk.rbd1.in‚0 disk.rbd2.out‚292.1611328125& disk.nvme0c0n1.out‚36519.435546875 disk.sda2.in‚0) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.52559371833699 disk.rbd1.out‚ 4.1669921875 disk.rbd3.in‚0 disk.rbd2.in‚0 cpu‚25.355443815083106 disk.drbd1777.in‚0( proc.memory.availableMB‚ 949.94921875# proc.memory.rssMB‚ 2875.96484375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 324.8359375% disk./.usageGB‚1110.4852027893066 disk.sda1.in‚0 disk.sda3.out‚ 324.8359375 proc.cpu.threads‚17‘„ü“é&‚(:ˆ ÏÙæÔ…è²Õ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.3458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬:*EKê&‚(:À ÏÙæÔ—¬ðØ% disk./.usageGB‚1110.4852027893066 disk.rbd1.in‚0# disk.dm-0.out‚36521.87548828125 disk.sda2.in‚0$ memory_percent‚94.19679641723633& proc.memory.availableMB‚ 950.796875 disk.drbd1777.in‚0 disk.sda3.out‚ 325.58984375 disk.dm-1.out‚ 325.58984375) proc.memory.percent‚17.55349636077881 network.recv‚ 2.999767e+06 disk.rbd1.out‚ 4.22998046875 disk.sda.out‚ 325.58984375 disk.sda3.in‚ 0.01171875 network.sent‚ 2.5809135e+07 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 1536.453125& disk.nvme0n1.out‚36521.87548828125 disk.drbd1777.out‚0) disk./.usagePercent‚74.52559371833699 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚36521.87548828125 disk.dm-1.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375 disk.sda.in‚ 0.01171875 disk.rbd2.out‚292.2666015625 disk.nvme0n1.in‚ 1536.453125 disk.dm-0.in‚ 1536.453125 cpu‚25.354861174841655 proc.cpu.threads‚17ö¶Œ2ë&- _wandbruntime‚8249  _runtime‚8249;ánìÈì& % _timestamp‚1.7884561578166928e+09 _runtime‚8249.371133781 _step‚868 & lr-AdamW/pg1‚8.406760218017037e-06 & lr-AdamW/pg2‚8.406760218017037e-06 trainer/global_step‚10599lÕbäÍí&Ç _step‚868 & lr-AdamW/pg1‚8.406760218017037e-06 & lr-AdamW/pg2‚8.406760218017037e-06 trainer/global_step‚10599 % _timestamp‚1.7884561578166928e+09 _runtime‚8249.371133781äj/'Ú2î&- _wandbruntime‚8249  _runtime‚8249yÓEÓè'ï&âO _runtime‚8249.372135257 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200013142 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884561578175545e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307755298304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599787539145728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615921440786944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154023422 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624795014831616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8840235863500 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556041728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200033180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8840890301256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615921458789888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672607844235776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8873556041728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50611533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969868 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚869 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154023420 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 354056602 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50611090 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615921458789888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681915758963712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9307914727936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140476051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89864516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608628429446984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414meÚçð&çO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9307755298304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599787539145728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615921440786944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.smal´ôm%l_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154023422 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624795014831616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8840235863500 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556041728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200033180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8840890301256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615921458789888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672607844235776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8873556041728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50611533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969868 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚869 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154023420 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 354056602 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50611090 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615921458789888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681915758963712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9307914727936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140476051 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89864516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608628429446984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚8249.372135257 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200013142 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884561578175545e+09å\Mežñ&‚(:“ ÞÙæÔËÅáÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.809' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.561799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žäˆ›Jò&‚(:¿ ÞÙæÔ‡–ÚÙ disk.rbd1.out‚ 4.22998046875 disk.sda.in‚ 0.01171875 proc.cpu.threads‚17# disk.dm-0.out‚36619.64501953125 disk.dm-1.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.rbd0.out‚0$ memory_percent‚94.19736862182617 disk.dm-1.out‚ 326.19140625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 1536.453125& proc.memory.availableMB‚ 950.703125# proc.memory.rssMB‚ 2875.96484375) disk./.usagePercent‚74.52558987822397 disk.dm-0.in‚ 1536.453125 disk.rbd2.out‚293.8408203125( disk.nvme0c0n1.out‚36619.64501953125 network.sent‚ 2.5815374e+07% disk./.usageGB‚1110.4851455688477 network.recv‚ 3.004307e+06 disk.rbd2.in‚0 cpu‚25.35433352611312& disk.nvme0n1.out‚36619.64501953125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 326.19140625) proc.memory.percent‚17.55349636077881 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 326.19140625 disk.nvme0n1.in‚ 1536.453125 disk.sda2.in‚0 disk.rbd1.in‚0àu;2ó&- _wandbruntime‚8250  _runtime‚82505½¦ò&ô&ìM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615977811828736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719107072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200031596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672669471033344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50615666 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9308774719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚870 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978269692928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978075850752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624852187311616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375482880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50616139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8874375482880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308604817408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154037438 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140488350 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608683478535392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8250.095587239 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884561585406828e+09<ÄÏ÷&õ&ñM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200031596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672669471033344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50615666 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9308774719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚870 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978269692928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978075850752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624852187311616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375482880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50616139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8874375482880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308604817408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154037438 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140488350 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608683478535392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8250.095587239 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884561585406828e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615977811828736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719107072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50839127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9351087515648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350925988864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 % _timestamp‚1.7884561940893767e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141112043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685041468634624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939194 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141112476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041283167744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939194 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675690381118976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430¡Šì'‚'æO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141112043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685041468634624 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200939194 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141112476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685041283167744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939194 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675690381118976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _step‚873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚618739092729344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885485056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50839127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9351087515648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350925988864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 % _timestamp‚1.7884561940893767e+09éÉÝ& 2ƒ'- _wandbruntime‚8286  _runtime‚8286°QþmÖ„'êM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200957648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788456194759176e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825059840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075836 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825059840 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154738236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50843607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675752031856640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361288704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351788636672 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 X »ä JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611470191880124 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141124691 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355695884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103796553216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9351954344448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8914704912896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154738238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8881887637872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚618795463771136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _runtime‚ 8286.31398589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50844055 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1©}îõ&…'ïM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611470191880124 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141124691 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355695884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103796553216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9351954344448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8914704912896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154738238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8881887637872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚618795463771136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _runtime‚ 8286.31398589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154738238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50844055 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200957648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611469520402800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788456194759176e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825059840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075836 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825059840 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154738236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50843607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675752031856640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8915361288704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351788636672 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2êUDÔ2†'- _wandbruntime‚8286  _runtime‚8286¨ÀïZ‡'Ô 1loss/FBt1/assignment_loss‚0.02092342244577594 0loss/FBt4/assignment_loss‚0.1720607747217097 /loss/FBt3/detection_loss‚2.3718497497694835 % _timestamp‚1.7884561947607563e+09 /loss/FRt4/assignment_loss‚3.365225968360901 2loss/SRqqt4/assignment_loss‚1.4488307431872403 epoch‚8 _runtime‚ 8286.31504719 _step‚875 /loss/FRt3/assignment_loss‚2.696317750088712 /loss/FRt4/detection_loss‚3.1697507071495057 /loss/FBt2/detection_loss‚0.5215731348457634 0loss/FRt2/assignment_loss‚1.1684149711028389 .loss/FRt3/detection_loss‚2.104360465039598 /loss/FBt1/detection_loss‚2.6134797732035318 /loss/FRt2/detection_loss‚1.7038650732163503 2loss/SRqqt2/assignment_loss‚0.8792425667653319 /loss/FRt1/detection_loss‚2.1251867111358376 1loss/SRqqt1/assignment_loss‚1.326646358312036 0loss/FBt3/assignment_loss‚0.2598067583250148 %loss/total_loss‚39.83227315088474 0loss/FRt1/assignment_loss‚2.4313624048674547 2loss/SRqqt3/assignment_loss‚1.0540540759555168 0loss/FBt2/assignment_loss‚0.2538454462447384 0loss/SRqqt2/detection_loss‚1.062732703495221 1loss/SRqqt4/detection_loss‚2.1909318623088656 1loss/SRqqt1/detection_loss‚1.6935534548874085 1loss/SRqqt3/detection_loss‚1.9667436328199175 .loss/FBt4/detection_loss‚3.231514642635981 trainer/global_step‚10649®‘ÓJ_ˆ'Ù 1loss/SRqqt1/assignment_loss‚1.326646358312036 0loss/FBt3/assignment_loss‚0.2598067583250148 %loss/total_loss‚39.83227315088474 0loss/FRt1/assignment_loss‚2.4313624048674547 2loss/SRqqt3/assignment_loss‚1.0540540759555168 0loss/FBt2/assignment_loss‚0.2538454462447384 0loss/SRqqt2/detection_loss‚1.062732703495221 1loss/SRqqt4/detection_loss‚2.1909318623088656 1loss/SRqqt1/detection_loss‚1.6935534548874085 1loss/SRqqt3/detection_loss‚1.9667436328199175 .loss/FBt4/detection_loss‚3.231514642635981 trainer/global_step‚10649 1loss/FBt1/assignment_loss‚0.02092342244577594 0loss/FBt4/assignment_loss‚0.1720607747217097 /loss/FBt3/detection_loss‚2.3718497497694835 % _timestamp‚1.7884561947607563e+09 /loss/FRt4/assignment_loss‚3.365225968360901 2loss/SRqqt4/assignment_loss‚1.4488307431872403 epoch‚8 _runtime‚ 8286.31504719 _step‚875 /loss/FRt3/assignment_loss‚2.696317750088712 /loss/FRt4/detection_loss‚3.1697507071495057 /loss/FBt2/detection_loss‚0.5215731348457634 0loss/FRt2/assignment_loss‚1.1684149711028389 .loss/FRt3/detection_loss‚2.104360465039598 /loss/FBt1/detection_loss‚2.6134797732035318 /loss/FRt2/detection_loss‚1.7038650732163503 2loss/SRqqt2/assignment_loss‚0.8792425667653319 /loss/FRt1/detection_loss‚2.1251867111358376ëI]è)ž‰'‚(:“ ‹ÚæÔé©ÝÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.254' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.450799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¸¢WMŠ'‚(: ‹ÚæÔû´ËØ disk.dm-1.in‚ 0.01171875( proc.memory.availableMB‚ 950.68359375 network.recv‚ 3.021007e+06) disk./.usagePercent‚74.52559960651027 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881 disk.drbd1777.in‚0 network.sent‚ 2.6014812e+07 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 327.78515625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚36820.2216796875$ memory_percent‚94.19748783111572% disk./.usageGB‚1110.4852905273438 disk.rbd1.out‚ 4.29296875 disk.sda.in‚ 0.01171875 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 disk.dm-1.out‚ 327.78515625 proc.cpu.threads‚17 disk.sda3.in‚ 0.01171875 disk.rbd2.out‚295.6767578125" disk.dm-0.out‚36820.2216796875 disk.sda2.out‚0# proc.memory.rssMB‚ 2875.96484375 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.sda.out‚ 327.78515625# disk.nvme0c0n1.in‚ 1560.38671875% disk.nvme0n1.out‚36820.2216796875 cpu‚25.352694611964292yñUç’‹'‚(:‡ šÚæÔóº× gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q¡',JŒ'‚(:¿ šÚæÔ‹·ïÙ disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875# disk.nvme0c0n1.in‚ 1560.38671875! disk.nvme0n1.in‚ 1560.38671875% disk.nvme0n1.out‚36820.4873046875$ memory_percent‚94.19841766357422 cpu‚25.352171845126875 disk.rbd1.out‚ 4.29296875 disk.rbd0.out‚0 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.01171875% proc.memory.availableMB‚ 950.53125# proc.memory.rssMB‚ 2875.96484375 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 328.42578125 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.52559576639726 disk.rbd3.in‚0 network.sent‚ 2.6123445e+07 network.recv‚ 3.027923e+06% disk./.usageGB‚1110.4852333068848' disk.nvme0c0n1.out‚36820.4873046875" disk.dm-0.out‚36820.4873046875 disk.dm-1.out‚ 328.42578125) proc.memory.percent‚17.55349636077881 disk.rbd1.in‚0 disk.sda.out‚ 328.42578125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.rbd2.out‚295.6767578125 disk.drbd1777.in‚0yƒàm2'- _wandbruntime‚8321  _runtime‚8321ÈõW ÈŽ' % _timestamp‚1.7884562298910584e+09 _runtime‚8321.445494073 _step‚876 & lr-AdamW/pg1‚7.331614719846282e-06 & lr-AdamW/pg2‚7.331614719846282e-06 trainer/global_step‚10699GÒ6OÍ'Ç trainer/global_step‚10699 % _timestamp‚1.7884562298910584e+09 _runtime‚8321.445494073 _step‚876 & lr-AdamW/pg1‚7.331614719846282e-06 & lr-AdamW/pg2‚7.331614719846282e-06솫‹2'- _wandbruntime‚8321  _runtime‚8321úß{3è'‘'âO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865246 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688166874724864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9394144286720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8955527653376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141750413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621556762674688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290328064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9393980662784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51069095 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10699 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚877 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201885284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9393980662784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51069095 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10699 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitoÿ{@5ør.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚877 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141750413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621556762674688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290328064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8922560980072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141762657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884562305622165e+09 _runtime‚8322.116990509 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415244 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8922722977280 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347094528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚605332839292928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚621613115713536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357322736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395003273728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690718720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621613097710592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141763104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155439038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155439036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605332856332288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630568788429312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 2019037644{C¶÷&•'ñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201903764 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357342802 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229572013568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462808064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141762657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614256233747532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884562305622165e+09 _runtime‚8322.116990509 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415244 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8922722977280 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚878 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347094528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚605332839292928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚621613115713536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357322736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9395003273728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8955690718720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621613097710592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚678834544799744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚621613097710592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141763104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155439038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155439036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605332856332288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630568788429312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0îY(2–'- _wandbruntime‚8322  _runtime‚8322&°Ú\—'Ö .loss/FRt1/detection_loss‚2.415922085504706 1loss/SRqqt4/detection_loss‚2.7143405310132285 _step‚879 /loss/FRt3/detection_loss‚2.1747472145491176 %loss/total_loss‚41.68078843470074 trainer/global_step‚10699 % _timestamp‚1.7884562305637264e+09 /loss/FBt2/detection_loss‚0.7824954192167967 0loss/FRt2/assignment_loss‚1.5535709752756006 2loss/SRqqt3/assignment_loss‚0.9694478061449315 2loss/SRqqt4/assignment_loss‚1.9175388506529005 /loss/FRt4/detection_loss‚3.0263894349336624 1loss/SRqqt1/detection_loss‚1.7870176169185006 /loss/FBt1/detection_loss‚2.8201511833402844 _runtime‚8322.118315184 0loss/FRt3/assignment_loss‚3.2865295288739382 2loss/SRqqt1/assignment_loss‚1.2864466090971718 2loss/SRqqt2/assignment_loss‚0.9866279028435001 1loss/SRqqt3/detection_loss‚1.5577793786568301 /loss/FBt3/detection_loss‚2.3655528724193573 .loss/FBt4/detection_loss‚2.539592057466507 /loss/FRt1/assignment_loss‚2.520412027835846 1loss/FBt2/assignment_loss‚0.17258979335857846 1loss/FBt1/assignment_loss‚0.08232758831258656 /loss/FBt3/assignment_loss‚0.732563988189213 0loss/FBt4/assignment_loss‚0.2148549676581723 epoch‚8 .loss/FRt2/detection_loss‚1.596689844759656 1loss/SRqqt2/detection_loss‚0.9201174481314117 /loss/FRt4/assignment_loss‚3.257083309548242›o‚6a˜'Û epoch‚8 .loss/FRt2/detection_loss‚1.596689844759656 1loss/SRqqt2/detection_loss‚0.9201174481314117 /loss/FRt4/assignment_loss‚3.257083309548242 .loss/FRt1/detection_loss‚2.415922085504706 1loss/SRqqt4/detection_loss‚2.7143405310132285 _step‚879 /loss/FRt3/detection_loss‚2.1747472145491176 %loss/total_loss‚41.68078843470074 trainer/global_step‚10699 % _timestamp‚1.7884562305637264e+09 /loss/FBt2/detection_loss‚0.7824954192167967 0loss/FRt2/assignment_loss‚1.5535709752756006 2loss/SRqqt3/assignment_loss‚0.9694478061449315 2loss/SRqqt4/assignment_loss‚1.9175388506529005 /loss/FRt4/detection_loss‚3.0263894349336624 1loss/SRqqt1/detection_loss‚1.7870176169185006 /loss/FBt1/detection_loss‚2.8201511833402844 _runtime‚8322.118315184 0loss/FRt3/assignment_loss‚3.2865295288739382 2loss/SRqqt1/assignment_loss‚1.2864466090971718 2loss/SRqqt2/assignment_loss‚0.9866279028435001 1loss/SRqqt3/detection_loss‚1.5577793786568301 /loss/FBt3/detection_loss‚2.3655528724193573 .loss/FBt4/detection_loss‚2.539592057466507 /loss/FRt1/assignment_loss‚2.520412027835846 1loss/FBt2/assignment_loss‚0.17258979335857846 1loss/FBt1/assignment_loss‚0.08232758831258656 /loss/FBt3/assignment_loss‚0.732563988189213 0loss/FBt4/assignment_loss‚0.2148549676581723ï/~%-’™'‚(:‡ ©ÚæÔߟÐÕ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.568' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÞ¡Hš'‚(:½ ©ÚæÔ£ÀÙ network.recv‚ 3.03196e+06 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875' disk.nvme0c0n1.out‚36918.3212890625 disk.drbd1777.in‚0' proc.memory.availableMB‚ 950.3671875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda1.in‚0$ memory_percent‚94.19941902160645$ disk./.usageGB‚1110.485523223877 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.rbd3.out‚0 cpu‚25.351604011728774 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.dm-1.out‚ 329.1796875) proc.memory.percent‚17.55349636077881 network.sent‚ 2.6129099e+07 disk.rbd0.in‚0% disk.nvme0n1.out‚36918.3212890625 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875" disk.dm-0.out‚36918.3212890625 disk.sda.out‚ 329.1796875 disk.sda1.out‚0# proc.memory.rssMB‚ 2875.96484375) disk./.usagePercent‚74.52561522296986 disk.sda3.out‚ 329.1796875! disk.rbd2.out‚295.84521484375 disk.rbd1.in‚0U…ù6ž›'‚(:“ ¸ÚæÔÞͨ× gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.826200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kb®ˆ?œ'‚(:´ ¸ÚæÔ¿£Ø' disk.nvme0c0n1.out‚37016.0166015625 disk.dm-0.in‚ 1560.38671875 disk.rbd1.out‚ 4.29296875 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 950.48046875% disk.nvme0n1.out‚37016.0166015625 disk.sda2.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda.out‚329.625 network.sent‚ 2.6238168e+07 network.recv‚ 3.039123e+06% disk./.usageGB‚1110.4855995178223 disk.sda1.out‚0" disk.dm-0.out‚37016.0166015625# proc.memory.rssMB‚ 2875.96484375) disk./.usagePercent‚74.52562034312055 disk.sda3.out‚329.625 disk.sda2.in‚0 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.in‚0$ memory_percent‚94.19872760772705 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-1.out‚329.625 cpu‚25.351098568357866! disk.rbd2.out‚295.84521484375'³L“'‚(:ˆ ÇÚæÔ¾ÙªÕ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.842' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñOÿ`Yž'‚(:¿ ÇÚæÔªŠêØ network.sent‚ 2.6243322e+07 disk.rbd2.in‚0 disk.drbk5yûñd1777.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 330.2109375 disk.rbd3.out‚0) disk./.usagePercent‚74.52562571927876 disk.rbd0.out‚0& disk.nvme0n1.out‚37016.47314453125 disk.sda1.out‚0) proc.memory.percent‚17.55349636077881 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 disk.sda3.out‚ 330.2109375$ memory_percent‚94.19879913330078 cpu‚25.35055592591278 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 330.2109375 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.01171875 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875% proc.memory.availableMB‚ 950.46875# proc.memory.rssMB‚ 2875.96484375! disk.rbd2.out‚297.49755859375( disk.nvme0c0n1.out‚37016.47314453125 network.recv‚ 3.043161e+06% disk./.usageGB‚1110.4856796264648 disk.dm-0.in‚ 1560.38671875# disk.dm-0.out‚37016.47314453125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875! disk.nvme0n1.in‚ 1560.38671875Ì$i2Ÿ'- _wandbruntime‚8356  _runtime‚8356ÇÓ£§Ê 'Ä ' lr-AdamW/pg1‚6.8204984650954005e-06 ' lr-AdamW/pg2‚6.8204984650954005e-06 trainer/global_step‚10749 % _timestamp‚1.7884562653384817e+09 _runtime‚8356.892857089 _step‚880G¥UÏ¡'É _runtime‚8356.892857089 _step‚880 ' lr-AdamW/pg1‚6.8204984650954005e-06 ' lr-AdamW/pg2‚6.8204984650954005e-06 trainer/global_step‚10749 % _timestamp‚1.7884562653384817e+09ðˆ²åe2¢'- _wandbruntime‚8356  _runtime‚8356úV{ ç'£'áO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚681855430945280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156125822 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358917118 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156125822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202791298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51299350 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436958490112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292570707968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681855454885376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156125820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51298895 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚608023141690368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292389435392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚881 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092Å´@Àì'¤'æO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚608023141690368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292389435392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚881 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10749 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚681855430945280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156125822 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358917118 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156125822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202791298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51299350 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436958490112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292570707968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚681855454885376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156125820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51298895 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049208ñ£xïm2¥'- _wandbruntime‚8357  _runtime‚8357—ÜÍñ&¦'ëM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8997332900352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633428100556288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332900352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8357.560588743 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142401394 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788456266005385e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676524544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚608078062860288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51303510 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51303949 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614940 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355088739328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142401835 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437983116288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Æ!$LÖ§'ðM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8997332900352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633428100556288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624430767655936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332900352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8357.560588743 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142401394 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788456266005385e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676524544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚608078062860288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51303510 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51303949 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614940 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691355088739328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142401835 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437983116288ò!Œ£2¨'- _wandbruntime‚8357  _runtime‚8357ù0 Ä\©'Ö 0loss/FRt2/assignment_loss‚1.3850590384231423 2loss/SRqqt1/assignment_loss‚1.3783017647940488 0loss/FBt4/assignment_loss‚0.2641136774327606 2loss/SRqqt2/assignment_loss‚0.9079529515175677 1loss/SRqqt2/detection_loss‚1.0056725476326338 1loss/SRqqt3/detection_loss‚1.9572743432862418 /loss/FBt4/detection_loss‚2.9334027071793876 0loss/FRt4/assignment_loss‚3.8914379314942793 1loss/FBt2/assignment_loss‚0.17980763782663114 /loss/FBt2/detection_loss‚0.6362544684400323 1loss/SRqqt4/assignment_loss‚1.854001326141534 /loss/FRt1/detection_loss‚2.3453732515158863 /loss/FRt2/detection_loss‚1.5070203417244386 %loss/total_loss‚42.52731842215875 0loss/FBt3/assignment_loss‚0.1257403086638078 1loss/SRqqt4/detection_loss‚2.6629928703661316 .loss/FBt1/detection_loss‚3.208450174331665 epoch‚8 /loss/FRt3/assignment_loss‚3.239569192029992 2loss/SRqqt3/assignment_loss‚1.2380863501481256 trainer/global_step‚10749 $ _timestamp‚1.788456266007239e+09 _step‚883 1loss/FBt1/assignment_loss‚0.21365491681499407 /loss/FRt3/detection_loss‚1.9513679894865943 1loss/SRqqt1/detection_loss‚1.4264758226890413 /loss/FBt3/detection_loss‚2.2221520505845547 /loss/FRt1/assignment_loss‚2.853031425372414 .loss/FRt4/detection_loss‚3.140125334262848 _runtime‚8357.561640124‰%½»aª'Û trainer/global_step‚10749 $ _timestamp‚1.788456266007239e+09 _step‚883 1loss/FBt1/assignment_loss‚0.21365491681499407 /loss/FRt3/detection_loss‚1.9513679894865943 1loss/SRqqt1/detection_loss‚1.4264758226890413 /loss/FBt3/detection_loss‚2.2221520505845547 /loss/FRt1/assignment_loss‚2.853031425372414 .loss/FRt4/detection_loss‚3.140125334262848 _runtime‚8357.561640124 0loss/FRt2/assignment_loss‚1.3850590384231423 2loss/SRqqt1/assignment_loss‚1.3783017647940488 0loss/FBt4/assignment_loss‚0.2641136774327606 2loss/SRqqt2/assignment_loss‚0.9079529515175677 1loss/SRqqt2/detection_loss‚1.0056725476326338 1loss/SRqqt3/detection_loss‚1.9572743432862418 /loss/FBt4/detection_loss‚2.9334027071793876 0loss/FRt4/assignment_loss‚3.8914379314942793 1loss/FBt2/assignment_loss‚0.17980763782663114 /loss/FBt2/detection_loss‚0.6362544684400323 1loss/SRqqt4/assignment_loss‚1.854001326141534 /loss/FRt1/detection_loss‚2.3453732515158863 /loss/FRt2/detection_loss‚1.5070203417244386 %loss/total_loss‚42.52731842215875 0loss/FBt3/assignment_loss‚0.1257403086638078 1loss/SRqqt4/detection_loss‚2.6629928703661316 .loss/FBt1/detection_loss‚3.208450174331665 epoch‚8 /loss/FRt3/assignment_loss‚3.239569192029992 2loss/SRqqt3/assignment_loss‚1.2380863501481256óÞ>g&ž«'‚(:“ ÖÚæÔ݈èÕ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘*_NK¬'‚(:À ÖÚæÔ²î»Ø disk.sda.in‚ 0.01171875 disk.sda1.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.out‚0) proc.memory.percent‚17.55349636077881 disk.dm-1.in‚ 0.01171875 disk.rbd1.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚37114.01220703125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1560.38671875& disk.nvme0n1.out‚37114.01220703125 disk.sda2.in‚0 network.recv‚ 3.047958e+06! disk.rbd2.out‚297.49755859375 disk.drbd1777.in‚0$ memory_percent‚94.19827461242676 disk.rbd0.in‚0# proc.memory.rssMB‚ 2875.96484375 cpu‚25.34999133608593 disk.sda3.out‚ 330.5390625% disk./.usageGB‚1110.4853477478027 disk.rbd3.out‚0 disk.sda.out‚ 330.5390625 disk.rbd0.out‚0# disk.dm-0.out‚37114.01220703125 network.sent‚ 2.6249668e+07( disk./.usagePercent‚74.5256034466233 disk.sda3.in‚ 0.01171875 disk.dm-0.in‚ 1560.38671875 disk.dm-1.out‚ 330.5390625' proc.memory.availableMB‚ 950.5546875 disk.sda2.out‚0 disk.rbd1.out‚ 4.29296875Nº’­'‚(:‡ åÚæÔèàÕÕ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž‘»L®'‚(:Á åÚæÔˆÊ¦Ø network.recv‚ 3.05467e+06# memory_percent‚94.1976547241211 disk.dm-1.out‚ 331.14453125 disk.rbd0.in‚0 disk.drbd1777.in‚0# disk.dm-0.out‚37212.09326171875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1110.4853286743164 disk.sda.out‚ 331.14453125! disk.nvme0n1.in‚ 1560.38671875 cpu‚25.349457381686097 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1560.38671875 disk.rbd1.in‚0) proc.memory.percent‚17.55349636077881 disk.rbd0.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 331.14453125 disk.rbd2.in‚0! disk.rbd2.out‚297.57568359375( disk.nvme0c0n1.out‚37212.09326171875 disk.drbd1777.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.52560216658561& disk.nvme0n1.out‚37212.09326171875 disk.sda2.out‚0# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.6357685e+07# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-1.in‚ 0.01171875% proc.memory.availableMB‚ 950.65625‹+—É2¯'- _wandbruntime‚8392  _runtime‚8392ÂÚ Ê°'Ä ' lr-AdamW/pg2‚6.3271631221552595e-06 trainer/global_step‚10799 % _timestamp‚1.7884563012919736e+09 _runtime‚8392.846450129 _step‚884 ' lr-AdamW/pg1‚6.3271631221552595e-06Ôçóoϱ'É % _timestamp‚1.7884563012919736e+09 _runtime‚8392.846450129 _step‚884 ' lr-AdamW/pg1‚6.3271631221552595e-06 ' lr-AdamW/pg2‚6.3271631221552595e-06 trainer/global_step‚10799ôtd/2²'- _wandbruntime‚8392  _runtime‚8392Gÿœåè'³'âO _runtime‚8392.847246209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9036842902528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51529472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143027133 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619771925439500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143027590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51529017 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610768365257728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203717350 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚694418236152832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480084954624 % _timestamp‚1.7884563012926905e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203717350 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚694418052783104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚10799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565824512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480244384256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844D,ºã´'çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91498116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚885 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203717350 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚694418052783104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚10799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚636229565824512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480244384256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚8392.847246209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9036842902528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51529472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143027133 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚619771925439500 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143027590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51529017 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610768365257728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203717350ž‘ì  F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚694418236152832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480084954624 % _timestamp‚1.7884563012926905e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚610823269388288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9038318706176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203735804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619827646005024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094060 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360576440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91506286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚8393.515054489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738304512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156840638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203755868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684999642506240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10799 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480590012416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9480971446272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91506284 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627248419598336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9481120376832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143039851 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51534012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143040298 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚886 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884563019603097e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662330368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0fxá÷&·'ñM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480590012416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9480971446272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91506284 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627248419598336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9481120376832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143039851 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51534012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143040298 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚886 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884563019603097e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9037662330368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚610823269388288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9038318706176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203735804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619827646005024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094060 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360576440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91506286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610823252348928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚8393.515054489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚636286738304512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156840638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156840636 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203755868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684999642506240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10799 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10880 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚691849263198720 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚158 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9088540854236 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 205612507 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚626088421993436 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 158505428 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9568952499200 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚633617669153280 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52126934 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 205596530 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9122126087168 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚8499.380257929 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884564078254013e+09 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚92486588 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚616999881139200 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101956 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 144613682 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚626088929947188 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚642739795240448 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚633617687156224 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9122126087168 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 158505426 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10880 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚691849263198720 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.curren9E?WÞt‚23940096 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚158 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9088540854236 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 205612507 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚626088421993436 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 158505428 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9568952499200 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚633617669153280 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52126934 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 205596530 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9122126087168 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚8499.380257929 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 % _timestamp‚1.7884564078254013e+09 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚92486588 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0ü²óv—2Ø'- _wandbruntime‚8500  _runtime‚8500Œ»óÛY)Ù'ÓR TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚158 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 % _timestamp‚1.7884564087836294e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501229056 _step‚893 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚633654517185024 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚617035191938048 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 364136043 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 158516234 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120064 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 158516232 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚626124587316084 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 144624513 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚642777501229056 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 205619809 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 144624673 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚701456432617984 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8500.338385019 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚633654499182080 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 364136043 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚160 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚52131579 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚92493094 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463644 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 205603832 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9089395378036 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9122984044032 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚642776990395904 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 158516232 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9122984044032 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9569317625856 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0Âä^)Ú'ØR \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚633654499182080 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9122491213824 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 205603832 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚10880 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9122491213824 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚642776990395904 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701456571901952 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 158516232 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚626124587316084 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 144624513 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚642777501229056 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 205619809 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 144624673 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚701456432617984 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8500.338385019 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚633654499182080 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 364136043 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚160 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚52131579 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚92493094 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463644 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 205603832 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9089395378036 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9122984044032 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚642776990395904 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 158516232 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9122984044032 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9569317625856 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚158 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 % _timestamp‚1.7884564087836294e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501229056 _step‚893 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚633654517185024 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚617035191938048 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 364136043 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 158516234 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120064 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 B&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5094868540763855 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3058065176010132 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1596292108297348 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2031959593296051 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0857236459851265 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04197454825043678 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4113802909851074 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5121378302574158 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18293727934360504 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11324086785316467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9756624102592468 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2885083556175232 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156822919845581 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995273113250732 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6160714030265808 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9298220276832581 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44564566016197205 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11710093915462494 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08744855970144272 *jet/accuracy_2_of_4‚0.3981764316558838 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Puîøi±|rity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45457521080970764 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4150967597961426 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870854616165161 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5628175139427185 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_5‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17129527032375336 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6453447937965393 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15373565256595612 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_1_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.25449714064598083 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.589342474937439 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.021658988669514656 +particle/sensitivity‚0.3378022313117981 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5260782837867737 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.4524983763694763 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7291666865348816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3482021391391754 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890468716621399 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9821417927742004 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9889312982559204 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.452826589345932 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4110119044780731 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9842439889907837 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.21153846383094788 particle/accuracy_4_of_6‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49786847829818726 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666865348816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24683906137943268 +jet/accuracy_2_of_6‚0.49358975887298584 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 particle/accuracy_6_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8840643763542175 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5973255038261414 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5622889995574951 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14457224309444427 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1207602396607399 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23808921873569489 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6145131587982178 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9862947463989258 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004747368860989809 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02949964627623558 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1789747178554535 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09291188418865204 ,jet/accuracy_4_of_4‚0.005306611768901348 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35686060786247253 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013844797387719154 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5776329040527344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5459829568862915 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.518420398235321 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11464276164770126 +particle/specificity‚0.9515410661697388 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5593282580375671 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_4_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5139785408973694 jet/accuracy_6_of_6‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5095438957214355 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1003526896238327 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6112179160118103 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 (particle/f_score‚0.42219454050064087 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5556905269622803 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4426509141921997 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15938009321689606 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15624691545963287 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.372539758682251 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5145018696784973 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2953704297542572 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3159719705581665 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26808661222457886 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5730674862861633 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5671296715736389 jet/accuracy_1_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35992491245269775 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_5_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5342134833335876 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5322567820549011 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚10880 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0973588302731514 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 _step‚894 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2740510404109955 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.028382467105984688 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46725258231163025 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 *jet/accuracy_3_of_5‚0.1179339587688446 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03407659754157066 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46986475586891174 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5412530899047852 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18518708646297455 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5939354300498962 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27678436040878296 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6208333373069763 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_3_of_3‚0.025252729654312134 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.045279406011104584 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908857345581055 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33958330750465393 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5083582401275635 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1863275021314621 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6926810145378113 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04119602218270302 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5103448033332825 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5201820731163025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8557168245315552 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5758391618728638 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3522321581840515 particle/accuracy_1_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6007201671600342 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17339487373828888 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.197235569357872 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9691091775894165 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6015152335166931 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15700337290763855 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06607366353273392 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.444861501455307 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821170568466187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5459829568862915 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_4‚1 particle/accuracy_7_of_7‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040366411209 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9785522222518921 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1341029405593872 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5485788583755493 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08244048058986664 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5462962985038757 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.455050528049469 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14874021708965302 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2571183145046234 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3903830647468567 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978700876235962 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27063271403312683 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16631318628787994 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990445077419281 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6080325245857239 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.45967429876327515 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4243420958518982 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02427675388753414 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_3_of_4‚0.06933221966028214 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11023916304111481 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 )loss/val_total_loss‚42.22514239190783 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2809169590473175 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024693777784705162 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793760776519775 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5410153269767761 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22818194329738617 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 *jet/accuracy_1_of_2‚0.7740057110786438 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3462962508201599 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5791565179824829 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5864197611808777 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5738095641136169 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19123776257038116 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_3‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5782948136329651 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_3_of_6‚1 epoch‚8 % _timestamp‚1.7884564212610695e+09 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚8512.817276098 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10290162265300751 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019090743735432625 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12442678958177567 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2617165446281433 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977941513061523 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5693702697753906 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19256757199764252 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9896695017814636 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5012155175209045 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15621788799762726 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969764351844788 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877040386199951 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5479668378829956 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5293645262718201 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047978438436985016 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20525723695755005 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011511074379086494 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41566571593284607 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25039711594581604 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18518708646297455 particle/accuracy_3_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2013889104127884 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.571941614151001 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5437546968460083 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6170761585235596 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3507496118545532 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_3‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9610726833343506 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44627317786216736 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6110087037086487 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5473935008049011 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1587301641702652 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04501575231552124 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6005747318267822 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8640351295471191 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21401037275791168 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 jet/accuracy_6_of_7‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19782716035842896 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15231050550937653 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453476548194885 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 -jet/accuracy_5_of_5‚0.0026881720405071974 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.506944477558136 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22229354083538055 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14753186702728271 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5838598012924194ÛJ¼ Aß'Ÿ§ :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23808921873569489 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6145131587982178 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9862947463989258 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.00694444449ΘÓ0ù61845875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004747368860989809 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02949964627623558 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1789747178554535 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09291188418865204 ,jet/accuracy_4_of_4‚0.005306611768901348 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35686060786247253 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013844797387719154 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5776329040527344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5459829568862915 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.518420398235321 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11464276164770126 +particle/specificity‚0.9515410661697388 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5593282580375671 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_4_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5139785408973694 jet/accuracy_6_of_6‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5095438957214355 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1003526896238327 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6112179160118103 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 (particle/f_score‚0.42219454050064087 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5556905269622803 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4426509141921997 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15938009321689606 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15624691545963287 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.372539758682251 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5145018696784973 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2953704297542572 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3159719705581665 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26808661222457886 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5730674862861633 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5671296715736389 jet/accuracy_1_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35992491245269775 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_5_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5342134833335876 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5322567820549011 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚10880 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0973588302731514 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 _step‚894 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2740510404109955 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.028382467105984688 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.46725258231163025 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 *jet/accuracy_3_of_5‚0.1179339587688446 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03407659754157066 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46986475586891174 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5412530899047852 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18518708646297455 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5939354300498962 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27678436040878296 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6208333373069763 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_3_of_3‚0.025252729654312134 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.045279406011104584 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908857345581055 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33958330750465393 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5083582401275635 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1863275021314621 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6926810145378113 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04119602218270302 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5103448033332825 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5201820731163025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8557168245315552 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5758391618728638 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3522321581840515 particle/accuracy_1_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6007201671600342 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17339487373828888 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.197235569357872 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9691091775894165 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6015152335166931 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15700337290763855 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06607366353273392 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.444861501455307 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821170568466187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5459829568862915 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_4‚1 particle/accuracy_7_of_7‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040366411209 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9785522222518921 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1341029405593872 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5485788583755493 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08244048058986664 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5462962985038757 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.455050528049469 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14874021708965302 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2571183145046234 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3903830647468567 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978700876235962 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27063271403312683 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16631318628787994 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.990445077419281 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6080325245857239 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.45967429876327515 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4243420958518982 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02427675388753414 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_3_of_4‚0.06933221966028214 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.42500001192092896 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11023916304111481 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 )loss/val_total_loss‚42.22514239190783 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2809169590473175 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024693777784705162 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793760776519775 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5410153269767761 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22818194329738617 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 *jet/accuracy_1_of_2‚0.7740057110786438 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3462962508201599 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5791565179824829 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5864197611808777 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5738095641136169 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19123776257038116 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_3‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5782948136329651 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_3_of_6‚1 epoch‚8 % _timestamp‚1.7884564212610695e+09 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚8512.817276098 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10290162265300751 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019090743735432625 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12442678958177567 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2617165446281433 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977941513061523 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5693702697753906 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19256757199764252 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9896695017814636 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5012155175209045 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15621788799762726 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969764351844788 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9877040386199951 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5479668378829956 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5293645262718201 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047978438436985016 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20525723695755005 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011511074379086494 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41566571593284607 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25039711594581604 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18518708646297455 particle/accuracy_3_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2013889104127884 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.571941614151001 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5437546968460083 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6170761585235596 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3507496118545532 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_3‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9610726833343506 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44627317786216736 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6110087037086487 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5473935008049011 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1587301641702652 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04501575231552124 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6005747318267822 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8640351295471191 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21401037275791168 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 jet/accuracy_6_of_7‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19782716035842896 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15231050550937653 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453476548194885 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 -jet/accuracy_5_of_5‚0.0026881720405071974 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.506944477558136 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22229354083538055 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14753186702728271 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5838598012924194 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16185516119003296 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2004142701625824 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_4_of_6‚0.07692307978868484 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15302444994449615 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.130674347281456 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9865360856056213 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.443572998046875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5094868540763855 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3058065176010132 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1596292108297348 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2031959593296051 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0857236459851265 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04197454825043678 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4113802909851074 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5121378302574158 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18293727934360504 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11324086785316467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9756624102592468 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2885083556175232 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8156822919845581 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995273113250732 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6160714030265808 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9298220276832581 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44564566016197205 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11710093915462494 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08744855970144272 *jet/accuracy_2_of_4‚0.3981764316558838 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.45457521080970764 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4150967597961426 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870854616165161 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5628175139427185 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_5‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17129527032375336 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6453447937965393 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15373565256595612 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_1_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.25449714064598083 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.589342474937439 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.021658988669514656 +particle/sensitivity‚0.3378022313117981 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5260782837867737 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt0FBt/event_purityñmÂl‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.4524983763694763 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7291666865348816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3482021391391754 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890468716621399 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9821417927742004 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9889312982559204 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.452826589345932 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4110119044780731 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9842439889907837 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.21153846383094788 particle/accuracy_4_of_6‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49786847829818726 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666865348816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24683906137943268 +jet/accuracy_2_of_6‚0.49358975887298584 ( Purity/2FRt3SRqqt1FBt/FRt_purity‚0.5 particle/accuracy_6_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8840643763542175 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5973255038261414 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5622889995574951 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14457224309444427 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1207602396607399 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 ("Purity/4FRt1SRqqt0FBt/event_purity‚0þ/ø\’à'‚(:‡ ìÛæÔÓÑØÕ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚33.0 gpu.0.powerWatts‚153.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.689 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÀyJá'‚(:¿ ìÛæÔÑÐ×Ø disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 336.3046875 disk.rbd1.out‚ 4.29296875# disk.nvme0c0n1.in‚ 1560.38671875% proc.memory.availableMB‚ 1427.3125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.55349636077881# disk.dm-0.out‚38175.21337890625 disk.sda2.in‚0 disk.sda.out‚ 336.3046875 disk.sda1.in‚0 cpu‚25.38155596342849 disk.rbd3.out‚0$ memory_percent‚91.28837585449219 network.recv‚ 3.102328e+06) disk./.usagePercent‚74.52580671660549 disk.dm-0.in‚ 1560.38671875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚38175.21337890625! disk.nvme0n1.in‚ 1560.38671875 disk.sda1.out‚0 disk.rbd0.in‚0! disk.rbd2.out‚301.21337890625# proc.memory.rssMB‚ 2875.96484375 disk.sda.in‚ 0.01171875 disk.sda3.in‚ 0.01171875 disk.sda3.out‚ 336.3046875% disk./.usageGB‚1110.4883766174316& disk.nvme0n1.out‚38175.21337890625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 network.sent‚ 2.6718403e+07-uÒÑ2â'- _wandbruntime‚8530  _runtime‚8530çØüÈã' & lr-AdamW/pg1‚5.394242686150469e-06 & lr-AdamW/pg2‚5.394242686150469e-06 trainer/global_step‚10899 % _timestamp‚1.7884564385307693e+09 _runtime‚8530.085150339 _step‚895 õ‚Íä'Ç trainer/global_step‚10899 % _timestamp‚1.7884564385307693e+09 _runtime‚8530.085150339 _step‚895 & lr-AdamW/pg1‚5.394242686150469e-06 & lr-AdamW/pg2‚5.394242686150469e-06ÿÖ´¸Ë2å'- _wandbruntime‚8530  _runtime‚8530= k³è'æ'âO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158908998 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚423 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206031417 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078881089024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9589668283392 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158909000 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031417 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9142490846720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144996052 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490846720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158909000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031417 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9142490846720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144996052 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9142490846720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158909000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158923016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52275428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653912064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52275850 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9143310287872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310287872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141387032064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@Devò0è‡iceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703141580874240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693550863543296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145008264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9590693390848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884564392090168e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644347325585408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚627646832564990 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 _step‚897 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206069935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚8530.764016037XnÎÙ÷&ê'ñM _step‚897 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206069935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚8530.764016037 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635204015297536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9590523488768 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158923016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52275428 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚627646161087666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚635203997294592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚10899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653912064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52275850 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9143310287872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635203997294592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310287872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703141387032064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703141580874240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206049871 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693550863543296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158923014 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145008264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9590693390848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884564392090168e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644347325585408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚627646832564990 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 7mÿ2ë'- _wandbruntime‚8530  _runtime‚8530î1MXì'Ò /loss/FRt4/assignment_loss‚3.556888155937195 %loss/total_loss‚42.07480968451722 /loss/FRt1/assignment_loss‚2.598145203930991 /loss/FRt3/assignment_loss‚2.724379149248015 2loss/SRqqt4/assignment_loss‚1.8455800159889109 /loss/FRt4/detection_loss‚2.9537465381622314 1loss/SRqqt2/detection_loss‚0.9487729081440539 .loss/FBt2/detection_loss‚0.506666153178387 0loss/FRt2/assignment_loss‚1.6274736846907665 1loss/SRqqt1/assignment_loss‚1.373455994006879 2loss/SRqqt2/assignment_loss‚0.8578799231133113 .loss/FBt1/detection_loss‚3.178979776799679 1loss/FBt2/assignment_loss‚0.16885070004964875 0loss/FBt3/assignment_loss‚0.6807390153925452 -loss/FRt1/detection_loss‚2.83500725712095 /loss/FRt2/detection_loss‚1.5331631778913029 epoch‚9 % _timestamp‚1.7884564392107806e+09 2loss/SRqqt3/assignment_loss‚1.5283051317800647 /loss/FRt3/detection_loss‚1.9571868836317423 1loss/SRqqt3/detection_loss‚1.8954346447211246 0loss/SRqqt4/detection_loss‚2.244636090362773 trainer/global_step‚10899 _runtime‚8530.765072918 _step‚898 1loss/FBt1/assignment_loss‚0.00715693489422442 2loss/FBt4/assignment_loss‚0.026526570164908964 0loss/SRqqt1/detection_loss‚1.692386987777251 .loss/FBt3/detection_loss‚2.566981421576606 .loss/FBt4/detection_loss‚2.766467365953657{ Ô]í'× 2loss/SRqqt2/assignment_loss‚0.8578799231133113 .loss/FBt1/detection_loss‚3.178979776799679 1loss/FBt2/assignment_loss‚0.16885070004964875 0loss/FBt3/assignment_loss‚0.6807390153925452 -loss/FRt1/detection_loss‚2.83500725712095 /loss/FRt2/detection_loss‚1.5331631778913029 epoch‚9 % _timestamp‚1.7884564392107806e+09 2loss/SRqqt3/assignment_loss‚1.5283051317800647 /loss/FRt3/detection_loss‚1.9571868836317423 1loss/SRqqt3/detection_loss‚1.8954346447211246 0loss/SRqqt4/detection_loss‚2.244636090362773 trainer/global_step‚10899 _runtime‚8530.765072918 _step‚898 1loss/FBt1/assignment_loss‚0.00715693489422442 2loss/FBt4/assignment_loss‚0.026526570164908964 0loss/SRqqt1/detection_loss‚1.692386987777251 .loss/FBt3/detection_loss‚2.566981421576606 .loss/FBt4/detection_loss‚2.766467365953657 /loss/FRt4/assignment_loss‚3.556888155937195 %loss/total_loss‚42.07480968451722 /loss/FRt1/assignment_loss‚2.598145203930991 /loss/FRt3/assignment_loss‚2.724379149248015 2loss/SRqqt4/assignment_loss‚1.8455800159889109 /loss/FRt4/detection_loss‚2.9537465381622314 1loss/SRqqt2/detection_loss‚0.9487729081440539 .loss/FBt2/detection_loss‚0.506666153178387 0loss/FRt2/assignment_loss‚1.6274736846907665 1loss/SRqqt1/assignment_loss‚1.373455994006879‚„Àmžî'‚(:“ ûÛæÔŒ’ª× gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.512' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.902400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒgîÐKï'‚(:À ûÛæÔ–ìÍÙ disk.drbd1777.out‚0 disk.sda1.in‚0) disk./.usagePercent‚74.52581055671848& disk.nvme0n1.out‚38272.71826171875 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875) proc.memory.percent‚17.55349636077881 disk.drbd1777.in‚0 disk.sda3.out‚ 336.953125 cpu‚25.38099857982782 disk.sda.in‚ 0.01171875 disk.rbd2.in‚0# disk.dm-0.out‚38272.71826171875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 336.953125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚38272.71826171875 network.recv‚ 3.109484e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 4.29296875! disk.rbd2.out‚301.33837890625$ memory_percent‚91.28706455230713# proc.memory.rssMB‚ 2875.96484375% disk./.usageGB‚1110.4884338378906 disk.dm-0.in‚ 1560.38671875 disk.dm-1.out‚ 336.953125 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1560.38671875) proc.memory.availableMB‚ 1427.52734375 proc.cpu.threads‚17 network.sent‚ 2.6842018e+07! disk.nvme0n1.in‚ 1560.38671875 disk.sda1.out‚0ò8糓ð'‚(:ˆ ŠÜæÔš¡ÇÕ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÝ0³Nñ'‚(:à ŠÜæÔ™Ëè× disk.sda.out‚ 337.2890625 disk.dm-0.in‚ 1560.38671875 proc.cpu.threads‚17 disk.rbd1.out‚ 4.29296875 disk.sda1.out‚0 disk.sda3.out‚ 337.2890625 cpu‚25.380430611284552# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚38844.07373046875 network.sent‚ 2.6950756e+07) disk./.usagePercent‚74.52572940233013 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 337.2890625% disk./.usageGB‚1110.4872245788574 disk.sda.in‚ 0.01171875 disk.drbd1777.out‚0( proc.memory.availableMB‚ 1427.6484375 disk.rbd0.out‚0& disk.nvme0n1.out‚38844.07373046875 disk.drbd1777.in‚0 network.recv‚ 3.116488e+06 disk.sda1.in‚0! disk.rbd2.out‚301.35400390625! disk.nvme0n1.in‚ 1560.38671875# disk.dm-0.out‚38844.07373046875# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875$ memory_percent‚91.28632545471191ty2žò'‚(:“ ™ÜæÔ£§ÒÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'àžPó'‚(:Å ™ÜæÔú¯Õ× disk.rbd2.in‚0# disk.dm-0.out‚38845.09326171875 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda.out‚ 337.94140625 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-0.in‚ 1560.38671875 cpu‚25.37982548442156) disk./.usagePercent‚74.52573708255616% disk./.usageGB‚1110.4873390197754 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 337.94140625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0) proc.memory.percent‚17.55349636077881 disk.sda3.out‚ 337.94140625( disk.nvme0c0n1.out‚38845.09326171875 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚301.4443359375! disk.nvme0n1.in‚ 1560.38671875) proc.memory.availableMB‚ 1427.39453125 network.sent‚ 2.6956366e+07 network.recv‚ 3.120755e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚38845.09326171875 disk.sda2.in‚0$ memory_percent‚91.28787517547607# proc.memory.rssMB‚ 2875.96484375 disk.rbd3.in‚0 disk.rbd0.in‚0Hëw2ô'- _wandbruntime‚8565  _runtime‚8565{ñœÊõ'Ä ' lr-AdamW/pg1‚4.9548520123063885e-06 ' lr-AdamW/pg2‚4.9548520123063885e-06 trainer/global_step‚10949 % _timestamp‚1.7884564743222203e+09 _runtime‚8565.876626767 _step‚899` ›wÏö'É % _timestamp‚1.7884564743222203e+09 _runtime‚8565.876626767 _step‚899 ' lr-AdamW/pg1‚4.9548520123063885e-06 ' lr-AdamW/pg2‚4.9548520123063885e-06 trainer/global_step‚10949ƒDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145628936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204560523264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706204366667776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52495866 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159609800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145628495 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884564743229406e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52495427 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820290048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438\‡ù'çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93133070 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159609800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647148790853632 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 C;DeviceStatsMonitor.on_train_batch_start/segment.al½ô¤f%l.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9632786894336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206977507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145628936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706204560523264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706204366667776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52495866 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159609800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145628495 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884564743229406e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52495427 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9182820290048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944„Ξu2ú'- _wandbruntime‚8566  _runtime‚8566œiâò&û'ìM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184296093696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183639717888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93141238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206995987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206995987 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚706267055395840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145641141 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612366 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633454835712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚10949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296093696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621282378780672 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚638021667239936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145641586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159623816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884564749846215e+09 _runtime‚8566.539643455 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9633631029248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206975923 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚706266855262208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚696633400426496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52499905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414g^÷&ü'ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9633454835712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 trainer/global_step‚10949 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296093696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621282378780672 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚638021667239936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145641586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159623816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884564749846215e+09 _runtime‚8566.539643455 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9633631029248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206975923 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚706266855262208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚696633400426496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52499905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9184296093696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9183639717888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93141238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206995987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206995987 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚706267055395840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145641141 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612366…5Nôo2ý'- _wandbruntime‚8566  _runtime‚8566ðòþ$^þ'Ø 0loss/FRt3/assignment_loss‚2.6217435491085053 /loss/FRt3/detection_loss‚2.0613266365230083 /loss/FRt4/detection_loss‚2.8799793792493418 trainer/global_step‚10949 /loss/FRt1/assignment_loss‚2.529949225809263 1loss/FBt3/assignment_loss‚0.25924229153539413 1loss/SRqqt1/detection_loss‚1.6909349822887667 1loss/SRqqt3/detection_loss‚1.6487979304633642 0loss/FRt4/assignment_loss‚2.9679297786770444 0loss/FBt1/assignment_loss‚0.5386010170303053 /loss/FRt2/detection_loss‚1.8366174991242588 0loss/FRt2/assignment_loss‚1.3448603823781013 1loss/SRqqt1/assignment_loss‚1.311950229142827 1loss/SRqqt2/detection_loss‚1.0691368479902545 &loss/total_loss‚40.335336555110544 1loss/FBt2/assignment_loss‚0.14531759279186374 .loss/FBt1/detection_loss‚3.205680012702942 /loss/FBt2/detection_loss‚0.4623908128407686 .loss/FBt4/detection_loss‚2.641027772426605 $ _timestamp‚1.788456474986899e+09 _runtime‚ 8566.54120167 2loss/SRqqt3/assignment_loss‚1.4322180538846736 .loss/FRt1/detection_loss‚2.238615121692419 _step‚902 2loss/SRqqt4/assignment_loss‚1.9037147437532742 2loss/FBt4/assignment_loss‚0.043323224474443124 1loss/SRqqt4/detection_loss‚2.3243346909681954 /loss/FBt3/detection_loss‚2.4482860565185547 epoch‚9 2loss/SRqqt2/assignment_loss‚0.7293587237363681ÕÚcÿ'Ý .loss/FRt1/detection_loss‚2.238615121692419 _step‚902 2loss/SRqqt4/assignment_loss‚1.9037147437532742 2loss/FBt4/assignment_loss‚0.043323224474443124 1loss/SRqqt4/detection_loss‚2.3243346909681954 /loss/FBt3/detection_loss‚2.4482860565185547 epoch‚9 2loss/SRqqt2/assignment_loss‚0.7293587237363681 0loss/FRt3/assignment_loss‚2.6217435491085053 /loss/FRt3/detection_loss‚2.0613266365230083 /loss/FRt4/detection_loss‚2.8799793792493418 trainer/global_step‚10949 /loss/FRt1/assignment_loss‚2.529949225809263 1loss/FBt3/assignment_loss‚0.25924229153539413 1loss/SRqqt1/detection_loss‚1.6909349822887667 1loss/SRqqt3/detection_loss‚1.6487979304633642 0loss/FRt4/assignment_loss‚2.9679297786770444 0loss/FBt1/assignment_loss‚0.5386010170303053 /loss/FRt2/detection_loss‚1.8366174991242588 0loss/FRt2/assignment_loss‚1.3448603823781013 1loss/SRqqt1/assignment_loss‚1.311950229142827 1loss/SRqqt2/detection_loss‚1.0691368479902545 &loss/total_loss‚40.335336555110544 1loss/FBt2/assignment_loss‚0.14531759279186374 .loss/FBt1/detection_loss‚3.205680012702942 /loss/FBt2/detection_loss‚0.4623908128407686 .loss/FBt4/detection_loss‚2.641027772426605 $ _timestamp‚1.788456474986899e+09 _runtime‚ 8566.54120167 2loss/SRqqt3/assignment_loss‚1.4322180538846736†2§bT€(‚(:“ ¨ÜæÔãóÕ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.877' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.375400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚15‡qÞIJ93! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%y²I(‚(:¾ ¨ÜæÔ²óÚØ! disk.nvme0n1.in‚ 1560.38671875 disk.sda3.out‚ 338.265625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.01171875 disk.rbd1.out‚ 4.29296875# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚38942.59326171875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1427.41796875# disk.dm-0.out‚38942.59326171875 disk.dm-1.out‚ 338.265625 disk.rbd2.out‚301.4599609375) proc.memory.percent‚17.55349636077881 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.6962545e+07# disk./.usageGB‚1110.48726272583 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚38942.59326171875 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2875.96484375 cpu‚25.379244123760103 network.recv‚ 3.124669e+06 disk.sda.in‚ 0.01171875 disk.sda1.out‚0 disk.rbd3.out‚0$ memory_percent‚91.28773212432861) disk./.usagePercent‚74.52573196240547 disk.sda.out‚ 338.265625 disk.rbd0.out‚0ýe»ž‚(‚(:“ ·ÜæÔÇïÒÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.669' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.733800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û¬ôKƒ(‚(:À ·ÜæÔ¼‡ç×$ disk./.usageGB‚1110.487319946289 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚91.28439426422119 cpu‚25.37870582218618& disk.nvme0n1.out‚39040.50634765625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚301.5849609375( disk.nvme0c0n1.out‚39040.50634765625 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1427.96484375) proc.memory.percent‚17.55349636077881 disk.sda.out‚ 338.9609375 disk.sda3.in‚ 0.01171875! disk.nvme0n1.in‚ 1560.38671875 proc.cpu.threads‚17# disk.dm-0.out‚39040.50634765625 disk.dm-1.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd0.out‚0 disk.rbd1.out‚ 4.29296875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2875.96484375) disk./.usagePercent‚74.52573580251848 disk.sda2.out‚0 disk.dm-1.out‚ 338.9609375 disk.rbd3.out‚0 network.recv‚ 3.13093e+06 disk.dm-0.in‚ 1560.38671875 disk.sda1.in‚0 disk.sda3.out‚ 338.9609375 disk.drbd1777.in‚0 network.sent‚ 2.7069985e+07 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1560.38671875q^2„(- _wandbruntime‚8601  _runtime‚86012§”ºÈ…( _runtime‚8601.312866451 _step‚903 & lr-AdamW/pg1‚4.533631048197522e-06 & lr-AdamW/pg2‚4.533631048197522e-06 trainer/global_step‚10999 % _timestamp‚1.7884565097584758e+09ïÈz͆(Ç & lr-AdamW/pg1‚4.533631048197522e-06 & lr-AdamW/pg2‚4.533631048197522e-06 trainer/global_step‚10999 % _timestamp‚1.7884565097584758e+09 _runtime‚8601.312866451 _step‚903‡a2‡(- _wandbruntime‚8601  _runtime‚8601Ù„ú?è'ˆ(âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9224462458368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884565097592356e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640782966143488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754236416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9675423667712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9223806095872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329899900928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚640782966143488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699654310512128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675589388800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734074 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516602 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚8601.313806893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52719266 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160310600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462458368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146260312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10999 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640782948140544 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚8601.313806893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623972681178112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52719266 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52718844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160310600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462458368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146260312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10999 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640782948140544 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329899900928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚640782966143488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699654310512128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9675589388800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734074 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430ˆãòÑj2Š(- _wandbruntime‚8601  _runtime‚8601Å~†«0#‹(ìM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224625523712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8601.993611166 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624027585308672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513810 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52723283 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699715961249792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9225281899520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272919 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚905 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922039 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224625523712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9191313951774 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640839301179392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926703104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601081856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9676447115264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_eÄIÆÂnd/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650064601081856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9676447115264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224625523712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8601.993611166 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624027585308672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513810 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52723283 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699715961249792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9225281899520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146272919 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚905 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922039 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224625523712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9191313951774 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336‰3»p2(- _wandbruntime‚8601  _runtime‚8601gDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652866066350080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391984640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9718454364160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626717904745472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11049 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455301759488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626717887706112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146892496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868óÎrbP™(çO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391984640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9718454364160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626717904745472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11049 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161011400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712455301759488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626717887706112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146892496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643600618085888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146892936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265448264192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9718286545920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418254 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 TDDeviceStatsMonitor.on_t?ò×rain_batch_start/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652866066350080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52942628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944Œîª=#2š(- _wandbruntime‚8638  _runtime‚8638讨üò&›(ìM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267705344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167214 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52947022 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828027 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884565464605532e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93958036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712517629952512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238830080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719315175936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161025416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93958038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161025414 _runtime‚8638.015433937 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146905058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚909P¨þ÷&œ(ñM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _step‚909 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9266267705344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167214 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52947022 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828027 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884565464605532e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93958036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712517629952512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238830080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9266267705344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9719315175936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161025416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208848091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93958038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161025414 _runtime‚8638.015433937 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146905058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430kW92(- _wandbruntime‚8638  _runtime‚8638¤BLXž(Ò /loss/FRt3/assignment_loss‚2.867066576713469 1loss/SRqqt1/detection_loss‚1.4063970951134699 .loss/FBt1/detection_loss‚3.066069326230458 1loss/SRqqt2/detection_loss‚1.0933351351602658 $ _timestamp‚1.788456546462139e+09 &loss/total_loss‚40.017910258403546 /loss/FRt2/assignment_loss‚1.012459790164774 0loss/FBt1/assignment_loss‚0.4432608868436075 /loss/FRt4/detection_loss‚3.0884440945727483 1loss/SRqqt3/detection_loss‚1.9830171620728922 0loss/FRt4/assignment_loss‚3.0350739870752608 2loss/SRqqt1/assignment_loss‚1.4720007922304303 2loss/SRqqt2/assignment_loss‚0.8079065933313875 2loss/SRqqt3/assignment_loss‚0.9724898178920186 1loss/SRqqt4/assignment_loss‚1.932406931919487 .loss/FRt2/detection_loss‚1.562081582674926 /loss/FBt4/detection_loss‚2.2081934412320456 _runtime‚8638.016733486 1loss/FBt2/assignment_loss‚0.11225114418261552 /loss/SRqqt4/detection_loss‚2.32345502470669 /loss/FBt3/detection_loss‚2.3161208480596542 _step‚910 trainer/global_step‚11049 1loss/FBt3/assignment_loss‚0.45903179406498634 /loss/FBt4/assignment_loss‚0.521071718656458 /loss/FRt1/detection_loss‚1.9982894820471604 .loss/FRt3/detection_loss‚2.115312297896641 /loss/FBt2/detection_loss‚0.5367954423322585 /loss/FRt1/assignment_loss‚2.685379293229845 epoch‚9#¸r©]Ÿ(× &loss/total_loss‚40.017910258403546 /loss/FRt2/assignment_loss‚1.012459790164774 0loss/FBt1/assignment_loss‚0.4432608868436075 /loss/FRt4/detection_loss‚3.0884440945727483 1loss/SRqqt3/detection_loss‚1.9830171620728922 0loss/FRt4/assignment_loss‚3.0350739870752608 2loss/SRqqt1/assignment_loss‚1.4720007922304303 2loss/SRqqt2/assignment_loss‚0.8079065933313875 2loss/SRqqt3/assignment_loss‚0.9724898178920186 1loss/SRqqt4/assignment_loss‚1.932406931919487 .loss/FRt2/detection_loss‚1.562081582674926 /loss/FBt4/detection_loss‚2.2081934412320456 _runtime‚8638.016733486 1loss/FBt2/assignment_loss‚0.11225114418261552 /loss/SRqqt4/detection_loss‚2.32345502470669 /loss/FBt3/detection_loss‚2.3161208480596542 _step‚910 trainer/global_step‚11049 1loss/FBt3/assignment_loss‚0.45903179406498634 /loss/FBt4/assignment_loss‚0.521071718656458 /loss/FRt1/detection_loss‚1.9982894820471604 .loss/FRt3/detection_loss‚2.115312297896641 /loss/FBt2/detection_loss‚0.5367954423322585 /loss/FRt1/assignment_loss‚2.685379293229845 epoch‚9 /loss/FRt3/assignment_loss‚2.867066576713469 1loss/SRqqt1/detection_loss‚1.4063970951134699 .loss/FBt1/detection_loss‚3.066069326230458 1loss/SRqqt2/detection_loss‚1.0933351351602658 $ _timestamp‚1.788456546462139e+09Ž£„«=ž (‚(:“ äÜæÔŽàìÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.144200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ó³ŸF¡(‚(:» äÜæÔ馾Ø& disk.nvme0n1.out‚39139.42822265625 disk.rbd3.in‚0 disk.sda.in‚ 0.01171875) proc.memory.percent‚17.55349636077881# disk.nvme0c0n1.in‚ 1560.38671875# disk.dm-0.out‚39139.42822265625 disk.dm-1.out‚ 340.09375 disk.rbd1.out‚ 4.29296875 disk.sda3.out‚ 340.09375( disk.nvme0c0n1.out‚39139.42822265625% disk./.usageGB‚1110.4874267578125 disk.dm-0.in‚ 1560.38671875 disk.sda.out‚ 340.09375 disk.rbd0.out‚0( proc.memory.availableMB‚ 1427.0234375# proc.memory.rssMB‚ 2875.96484375 network.sent‚ 2.7190729e+07) disk./.usagePercent‚74.52574297072945! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.01171875 disk.rbd0.in‚0 network.recv‚ 3.14643e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd2.out‚303.2529296875$ memory_percent‚91.29014015197754 cpu‚25.376965839441933xÅõÅž¢(‚(:“ óÜæÔžÜÒÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.283' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.056599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X²-´Q£(‚(:Æ óÜæÔöâƒØ disk.rbd0.out‚0# disk.dm-0.out‚39237.10791015625 disk.sda3.out‚ 340.52734375$ disk./.usageGB‚1110.487434387207 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚39237.10791015625! disk.nvme0n1.in‚ 1560.38671875 disk.sda3.in‚ 0.01171875 network.recv‚ 3.152889e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.out‚0 disk.sda.in‚ 0.01171875 disk.sda1.out‚0$ memory_percent‚91.28801822662354! disk.rbd2.out‚303.34326171875 disk.dm-1.out‚ 340.52734375 disk.rbd1.in‚0& disk.nvme0n1.out‚39237.10791015625 disk.rbd1.out‚ 4.29296875 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1427.37109375# proc.memory.rssMB‚ 2875.96484375 cpu‚25.376403661638093 proc.cpu.threads‚17 network.sent‚ 2.7298648e+07 disk.sda.out‚ 340.52734375) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.52574348274452 disk.rbd0.in‚0 disk.dm-1.in‚ 0.01171875e®¦Þ“¤(‚(:ˆ ‚ÝæÔ° ñÕ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Ô8N¥(‚(:à ‚ÝæÔœˆŸØ) disk./.usagePercent‚74.52572709826232 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.01171875 disk.dm-1.out‚ 340.98828125) proc.memory.percent‚17.55349636077881 disk.sda3.out‚ 340.98828125 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd1.in‚0 network.sent‚ 2.7304906e+07 disk.rbd0.in‚0 disk.rbd1.out‚ 4.29296875 disk.rbd3.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1560.38671875 disk.rbd3.out‚0# memory_percent‚94.1826343536377 proc.cpu.threads‚17$ disk./.usageGB‚1110.487190246582 disk.rbd0.out‚0! disk.rbd2.out‚303.35888671875 disk.drbd1777.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 953.1171875 cpu‚25.375839305595537# disk.dm-0.out‚39334.54541015625 network.recv‚ 3.157128e+06( disk.nvme0c0n1.out‚39334.54541015625 disk.sda2.out‚0 disk.sda.out‚ 340.98828125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875& disk.nvme0n1.out‚39334.54541015625 disk.drbd1777.out‚0 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2875.96484375ö­s2¦(- _wandbruntime‚8673  _runtime‚8673œˆ}_ɧ(à $ _timestamp‚1.788456581614526e+09 _runtime‚8673.168911531 _step‚911 ' lr-AdamW/pg1‚3.7460455716218522e-06 ' lr-AdamW/pg2‚3.7460455716218522e-06 trainer/global_step‚11099K$_B¨(È ' lr-AdamW/pg2‚3.7460455716218522e-06 trainer/global_¶*~Œstep‚11099 $ _timestamp‚1.788456581614526e+09 _runtime‚8673.168911531 _step‚911 ' lr-AdamW/pg1‚3.7460455716218522e-06< È2©(- _wandbruntime‚8673  _runtime‚8673NÁ:è'ª(âO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209755663 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655724029732864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705819384278528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚8673.169915012 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚646418252025344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161712198 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468890 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209735625 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147526092 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704098304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580807243264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655724704098304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735625 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305777707520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9305777707520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147526516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚715580625970688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306434070016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚629463111273472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 % _timestamp‚1.7884565816153524e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761422964736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306434070016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚629463111273472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 % _timestamp‚1.7884565816153524e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9761422964736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704098304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580807243264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655724704098304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735625 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952¥‹2¬(- _wandbruntime‚8673  _runtime‚8673ür}ò&­(ìM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53172741 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762120887808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307253511168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚715643131963904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034622 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646474623067136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9762286595584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚638790328557298 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8673.881934177 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚913 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209774143 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161726216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480293 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚715643321611776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147538737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307253511168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚715643131963904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034622 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161726214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMCwk ßonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646474623067136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9762286595584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚638790328557298 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8673.881934177 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚913 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209774143 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161726216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371480293 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚715643321611776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147538737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161726216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53172741 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9762120887808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162413000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667481088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148158452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763513344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53392215 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718706434161152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53391784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9804512999424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210661677 _step‚916 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681715 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚718706250791424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162413000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚632208300762112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210661677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148158885 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040*ÿíF- »(çO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649235921970688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9804353569792 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534826 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341846528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162412998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162413000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658582667481088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148158452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763513344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53392215 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718706434161152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53391784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9804512999424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210661677 _step‚916 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681715 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚718706250791424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162413000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes§XDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708963547959296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148171539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚917 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11149 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805379949056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969998 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680131 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514326528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768951848448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768772686336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8709.390249184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424706 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210700195 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94774838 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514326528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768951848448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚718768772686336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚422 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8709.390249184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚641576370424706 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210700195 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708963547959296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148171539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _step‚917 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708963571899392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11149 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9805379949056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969998 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210680131 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661441979594752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9388405681664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587729 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405681664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163113800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8744.473242226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721831808499712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163113798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148791606 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53616536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644306691726590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634953507290112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979594752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95175068 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smallJ¡Ÿh×_pool.allocated‚ 9847350454784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884566529186687e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9387749319168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607767 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831623032832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 m£í'Ë(çO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587729 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405681664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163113800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8744.473242226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721831808499712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163113798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148791606 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53616536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644306691726590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634953507290112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚661441979594752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95175068 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9847350454784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884566529186687e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652053555910144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9387749319168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711984434104832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607767 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831623032832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661441979594752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9388405681664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401˜;.”‘Ì(‚(:† ÍÝæÔµ‚ùÕ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 6iüDÍ(‚(:¹ ÍÝæÔÝŒÿÙ' proc.memory.availableMB‚ 952.8828125# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 disk.sda.out‚ 343.41796875 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚39628.80810546875 disk.rbd2.out‚ 305.328125$ disk./.usageGB‚1110.488166809082 disk.rbd0.out‚0 disk.dm-0.in‚ 1560.38671875 disk.sda.in‚ 0.0234375 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.rbd3.out‚0 disk.sda3.out‚ 343.41796875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚74.52579263619108# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda2.out‚0# disk.dm-0.out‚39628.80810546875 network.sent‚ 2.753955e+07 disk.sda3.in‚ 0.0234375 disk.rbd2.in‚0 disk.drbd1777.out‚0# memory_percent‚94.1840648651123 cpu‚25.37307095898657 disk.dm-1.in‚ 0.0234375 network.recv‚ 3.183489e+06( disk.nvme0c0n1.out‚39628.80810546875 disk.dm-1.out‚ 343.41796875Sß±2Î(- _wandbruntime‚8745  _runtime‚8745.{½øñ&Ï(ëM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9848214950912 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721894131988480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚8745.157570121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9848047145984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚644363083769438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721894323733504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53620987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661499152074752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211626247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788456653602533e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374733997 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9388568747008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225122816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354655309406 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53620555 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163127814 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148803791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635008428460032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944àæ5¤Ð(ðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606183 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53620555 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163127814 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148803791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635008428460032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9848214950912 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721894131988480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚8745.157570121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9848047145984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚644363083769438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721894323733504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch„ƒfÿã_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚434 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53620987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661499152074752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211626247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788456653602533e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374733997 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9388568747008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚712046084842496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225122816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354655309406 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788456688411108e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚637698713818112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391487488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 _runtime‚8779.965725922 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328379 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942977536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9394019775886 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149425752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9890192550912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724957163538944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9428735124992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890358272000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213642 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647093405071114 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149426183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617342976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212533819 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53842713 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚724957353200128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2w.-÷ì'Ý(æO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9394019775886 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149425752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9890192550912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724957163538944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9428735124992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890358272000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9394674213642 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647093405071114 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149426183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617342976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212533819 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53842713 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚724957353200128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚654871225855488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚654871225855488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 trainer/global_step‚11249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788456688411108e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚637698713818112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391487488 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 _runtime‚8779.965725922 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328379 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942977536œePD2Þ(- _wandbruntime‚8780  _runtime‚8780°E«Qß(ìM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚647149125636846 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚654927578894336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9430210928640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.curŠàåà"rent‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚925 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163828616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚637753634988032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715128621725696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647148454159522 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9429554552832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8780.635029174 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149438772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210928640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53847134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891059704320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149438336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53846700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019681430016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789822976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725019864786432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚925 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163828616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚637753634988032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95591638 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚715128621725696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9395490648814 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212552299 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647148454159522 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9429554552832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚8780.635029174 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149438772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376360849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210928640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53847134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891059704320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚637753617948672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149438336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53846700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019681430016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664357789822976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚654927560891392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725019864786432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975271 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213459871 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657688877797888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580725760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54068587 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728082656013312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377293312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8815.641211613 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991Á|oé¸ 868 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚667159255091200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9933148142080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚11299 $ _timestamp‚1.788456724086674e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0û/ò_ì'í(æO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728082849868800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657688877797888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580725760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54068587 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728082656013312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377293312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8815.641211613 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991868 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚667159255091200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9933148142080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚11299 $ _timestamp‚1.788456724086674e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚640443937385472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255091200 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975271 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213459871 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657745212833792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54073036 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884567247656486e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚718211182548992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96000036 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9934175924224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164529414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657745230836736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150073074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504254 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213458287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54072595 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164529416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150072631 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145358473216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8816.320467067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718211158608896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _step‚929 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚667216427571200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884567247656486e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚718211182548992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96000036 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9934175924224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164529414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164529414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657745230836736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150073074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504254 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213458287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54072595 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164529416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150072631 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728145358473216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚8816.320467067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718211158608896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _step‚929 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚667216427571200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657745212833792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54073036 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 86822_Ð:Ôâ0928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¡!¿Ä2ñ(- _wandbruntime‚8816  _runtime‚8816È2‰\ò(Ö /loss/FRt2/detection_loss‚1.6194971148110926 /loss/FRt3/detection_loss‚2.2336835249876366 1loss/SRqqt3/detection_loss‚1.6640882661529617 /loss/FBt2/detection_loss‚0.5699740325124115 1loss/SRqqt1/detection_loss‚1.5788266683618228 /loss/FBt1/detection_loss‚2.2356181343396506 1loss/FBt1/assignment_loss‚0.14659467266271045 0loss/SRqqt4/detection_loss‚2.807593679144269 % _timestamp‚1.7884567247675796e+09 _runtime‚8816.321897672 0loss/FRt3/assignment_loss‚2.9782087420805907 1loss/SRqqt2/assignment_loss‚0.930046838692347 1loss/SRqqt4/assignment_loss‚1.684718034629311 1loss/FBt3/assignment_loss‚0.42401513573713245 -loss/FRt1/detection_loss‚2.25107031997214 /loss/FRt4/detection_loss‚3.2748444234883345 /loss/FBt3/detection_loss‚1.9739990644156933 2loss/FBt2/assignment_loss‚0.055196028279971526 trainer/global_step‚11299 2loss/SRqqt1/assignment_loss‚1.1185439630722007 1loss/SRqqt3/assignment_loss‚1.380870924093852 /loss/FBt4/assignment_loss‚0.202698132389092 /loss/FBt4/detection_loss‚2.6255363396235873 &loss/total_loss‚39.619756460861694 epoch‚9 _step‚930 /loss/FRt1/assignment_loss‚2.603847477030247 0loss/FRt2/assignment_loss‚1.3724541079998016 /loss/FRt4/assignment_loss‚2.975484737643489 1loss/SRqqt2/detection_loss‚0.9123460987413471V\­aó(Û 2loss/SRqqt1/assignment_loss‚1.1185439630722007 1loss/SRqqt3/assignment_loss‚1.380870924093852 /loss/FBt4/assignment_loss‚0.202698132389092 /loss/FBt4/detection_loss‚2.6255363396235873 &loss/total_loss‚39.619756460861694 epoch‚9 _step‚930 /loss/FRt1/assignment_loss‚2.603847477030247 0loss/FRt2/assignment_loss‚1.3724541079998016 /loss/FRt4/assignment_loss‚2.975484737643489 1loss/SRqqt2/detection_loss‚0.9123460987413471 /loss/FRt2/detection_loss‚1.6194971148110926 /loss/FRt3/detection_loss‚2.2336835249876366 1loss/SRqqt3/detection_loss‚1.6640882661529617 /loss/FBt2/detection_loss‚0.5699740325124115 1loss/SRqqt1/detection_loss‚1.5788266683618228 /loss/FBt1/detection_loss‚2.2356181343396506 1loss/FBt1/assignment_loss‚0.14659467266271045 0loss/SRqqt4/detection_loss‚2.807593679144269 % _timestamp‚1.7884567247675796e+09 _runtime‚8816.321897672 0loss/FRt3/assignment_loss‚2.9782087420805907 1loss/SRqqt2/assignment_loss‚0.930046838692347 1loss/SRqqt4/assignment_loss‚1.684718034629311 1loss/FBt3/assignment_loss‚0.42401513573713245 -loss/FRt1/detection_loss‚2.25107031997214 /loss/FRt4/detection_loss‚3.2748444234883345 /loss/FBt3/detection_loss‚1.9739990644156933 2loss/FBt2/assignment_loss‚0.055196028279971526 trainer/global_step‚11299¢P~gÖô(‚(:’ ˜ÞæÔ¢Š¡× gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.315399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó7?rKõ(‚(:À ˜ÞæÔ—ÏÙ& disk.nvme0n1.out‚39923.00927734375 cpu‚25.370308617507266 network.recv‚ 3.211485e+06% disk./.usageGB‚1110.4884910583496 disk.sda3.in‚ 0.0234375 disk.drbd1777.out‚0# memory_percent‚94.1857099533081# proc.memory.rssMB‚ 2875.96484375 disk.dm-1.in‚ 0.0234375 disk.sda.in‚ 0.0234375! disk.nvme0n1.in‚ 1560.38671875 disk.rbd3.in‚0 disk.sda.out‚ 345.8203125 disk.sda3.out‚ 345.8203125 disk.rbd0.out‚0 disk.rbd1.in‚0( disk./.usagePercent‚74.5258143968315 disk.dm-0.in‚ 1560.38671875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚39923.00927734375 disk.rbd3.out‚0( proc.memory.availableMB‚ 952.61328125) proc.memory.percent‚17.55349636077881# disk.dm-0.out‚39923.00927734375 disk.sda2.in‚0 disk.sda1.in‚0! disk.rbd2.out‚307.16064453125 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.4736328125 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7855404e+07 disk.sda2.out‚0 disk.dm-1.out‚ 345.8203125 disk.sda1.out‚0X²qö(‚(:’ §ÞæÔüåÑÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.51' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.101999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ÄÐB÷(‚(:· §ÞæÔùï¨Ù( disk./.usagePercent‚74.5257672914452! disk.rbd2.out‚307.28564453125! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 cpu‚25.36976526973385% disk./.usageGB‚1110.4877891540527 disk.rbd0.out‚0$ disk.nvme0n1.out‚40020.603515625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 346.265625 network.recv‚ 3.21777e+06 disk.sda2.out‚0 disk.dm-1.out‚ 346.265625 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚40020.603515625$ memory_percent‚94.18356418609619) proc.memory.percent‚17.55349636077881 disk.sda1.out‚0 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.53662109375( proc.memory.availableMB‚ 952.96484375 network.sent‚ 2.7962596e+07 disk.dm-1.in‚ 0.0234375 disk.sda.in‚ 0.0234375 proc.cpu.threads‚17! disk.dm-0.out‚40020.603515625 disk.sda3.in‚ 0.0234375 disk.sda3.out‚ 346.265625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2875.96484375 $[’ø(‚(:‡ ¶ÞæÔ­¬ÃÕ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÍ4Aù(‚(:¶ ¶ÞæÔ´ƒÙ% proc.memory.availableMB‚ 952.96875# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 cpu‚25.369197939775134 disk.dm-1.out‚ 346.890625 disk.rbd0.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd1.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.0234375 disk.sda.in‚ 0.0234375 disk.sda1.in‚0! disk.rbd2.out‚308.87548828125$ memory_percent‚94.18354034423828! disk.dm-0.out‚40020.830078125 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚40020.830078125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 346.890625 network.sent‚ 2.7968859e+07( disk./.usagePercent‚74.5257672914452% disk./.usageGB‚1110.4877891540527 disk.sda2.in‚0 disk.rbd1.out‚ 4.53662109375 disk.sda1.out‚0$ disk.nvme0n1.out‚40020.830078125 network.recv‚ 3.222251e+06 disk.drbd1777.in‚0 disk.sda.out‚ 346.890625 disk.sda3.in‚ 0.0234375øÉt&2ú(- _wandbruntime‚8851  _runtime‚8851`²õÛÊû(Ä ' lr-AdamW/pg1‚2.1007282975799623e-06 ' lr-AdamW/pg2‚2.1007282975799623e-06 trainer/global_step‚11349 % _timestamp‚1.7884567597074852e+09 _runtime‚8851.261896693 _step‚931ZV Ïü(É % _timestamp‚1.7884567597074852e+09 _runtime‚8851.261896693 _step‚931 ' lr-AdamW/pg1‚2.1007282975799623e-06 ' lr-AdamW/pg2‚2.1007282975799623e-06 trainer/global_step‚11349£<2ý(- _wandbruntime‚8851  _runtime‚8851ÕáIÕå'þ(ßO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9476344892458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54294743 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54294308 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976398306816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 # _timestamp‚1.78845675970824e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208467001344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328814 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 8851.26285554 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721232044754432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365885 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150695013 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363099136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976243071488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165216200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208287825920T¾ê'ÿ(äO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150695013 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660506529740288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9511363099136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9976243071488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165216200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731208287825920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9476344892458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54294743 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643189126874112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54294308 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9976398306816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 # _timestamp‚1.78845675970824e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731208467001344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328814 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚ 8851.26285554 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚660506511737344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721232044754432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214365885 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437¤¬ >2€)- _wandbruntime‚8851  _runtime‚8851QÏÎ)ëM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214384339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731270797777920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165230214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788456760375896e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214384339 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150707344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚8851.930949988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 >6DeviceStatsMonitor.on_train_batch_end/segment.all.p¤Kå¼Øeak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065319424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299371 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚933 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614553 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54298908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270987425792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404403 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9512182540288 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150707809 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977102285824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928¶ÄôXö&‚)ðM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚933 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614553 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54298908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270987425792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404403 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9512182540288 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150707809 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9977102285824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660562882779136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214384339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731270797777920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96408438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165230214 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788456760375896e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214384339 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150707344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚8851.930949988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670075065319424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299371 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¥Ø~§H2ƒ)- _wandbruntime‚8851  _runtime‚8851­Ôi…\„)Ö 0loss/FRt2/assignment_loss‚1.2288202819689897 2loss/SRqqt1/assignment_loss‚1.3225655047688634 /loss/FRt1/detection_loss‚2.4104551695611165 /loss/FBt1/detection_loss‚2.3120219045215182 % _timestamp‚1.7884567603777425e+09 0loss/FRt1/assignment_loss‚2.5471800384314163 1loss/SRqqt2/detection_loss‚0.8241275054216385 epoch‚9 _runtime‚8851.932083253 1loss/FBt2/assignment_loss‚0.11687094665657231 .loss/FBt4/detection_loss‚2.595714818347584 %loss/total_loss‚40.91062225666471 _step‚934 1loss/SRqqt2/assignment_loss‚1.016523688522478 1loss/FBt4/assignment_loss‚0.21581791108341844 /loss/FRt3/detection_loss‚1.8064299933618015 .loss/FRt4/detection_loss‚3.463218316435814 /loss/FBt2/detection_loss‚0.4277361711523524 0loss/SRqqt1/detection_loss‚1.584855987673456 /loss/FRt4/assignment_loss‚3.361696457862854 2loss/SRqqt4/assignment_loss‚1.8098370024691457 1loss/FBt1/assignment_loss‚0.48547601908891536 /loss/FBt3/detection_loss‚2.3177933480058397 trainer/global_step‚11349 1loss/SRqqt3/assignment_loss‚1.485555781594788 1loss/FBt3/assignment_loss‚0.18559365440160036 .loss/FRt2/detection_loss‚1.517629033216918 /loss/FRt3/assignment_loss‚3.354505479335785 1loss/SRqqt3/detection_loss‚1.6594386557117105 1loss/SRqqt4/detection_loss‚2.8607585870701335taí2a…)Û /loss/FRt4/assignment_loss‚3.361696457862854 2loss/SRqqt4/assignment_loss‚1.8098370024691457 1loss/FBt1/assignment_loss‚0.48547601908891536 /loss/FBt3/detection_loss‚2.3177933480058397 trainer/global_step‚11349 1loss/SRqqt3/assignment_loss‚1.485555781594788 1loss/FBt3/assignment_loss‚0.18559365440160036 .loss/FRt2/detection_loss‚1.517629033216918 /loss/FRt3/assignment_loss‚3.354505479335785 1loss/SRqqt3/detection_loss‚1.6594386557117105 1loss/SRqqt4/detection_loss‚2.8607585870701335 0loss/FRt2/assignment_loss‚1.2288202819689897 2loss/SRqqt1/assignment_loss‚1.3225655047688634 /loss/FRt1/detection_loss‚2.4104551695611165 /loss/FBt1/detection_loss‚2.3120219045215182 % _timestamp‚1.7884567603777425e+09 0loss/FRt1/assignment_loss‚2.5471800384314163 1loss/SRqqt2/detection_loss‚0.8241275054216385 epoch‚9 _runtime‚8851.932083253 1loss/FBt2/assignment_loss‚0.11687094665657231 .loss/FBt4/detection_loss‚2.595714818347584 %loss/total_loss‚40.91062225666471 _step‚934 1loss/SRqqt2/assignment_loss‚1.016523688522478 1loss/FBt4/assignment_loss‚0.21581791108341844 /loss/FRt3/detection_loss‚1.8064299933618015 .loss/FRt4/detection_loss‚3.463218316435814 /loss/FBt2/detection_loss‚0.4277361711523524 0loss/SRqqt1/detection_loss‚1.584855987673456¦\Æò’†)‚(:‡ ÅÞæÔ±—ËÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚90.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆuèE‡)‚(:º ÅÞæÔ·ý˜Ø disk.sda3.in‚ 0.0234375 disk.rbd1.out‚ 4.599609375# proc.memory.rssMB‚ 2875.96484375# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda1.in‚0 disk.sda3.out‚ 347.26171875! disk.nvme0n1.in‚ 1560.38671875 disk.dm-0.in‚ 1560.38671875& proc.memory.availableMB‚ 952.984375! disk.rbd2.out‚309.00048828125 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 347.26171875 disk.rbd1.in‚0 disk.dm-1.in‚ 0.0234375 disk.rbd3.in‚0$ memory_percent‚94.18344497680664) proc.memory.percent‚17.55349636077881& disk.nvme0c0n1.out‚40118.447265625) disk./.usagePercent‚74.52579596428905 disk.rbd0.out‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚40118.447265625 disk.drbd1777.out‚0! disk.dm-0.out‚40118.447265625 proc.cpu.threads‚17 network.recv‚ 3.226438e+06% disk./.usageGB‚1110.4882164001465 disk.sda2.in‚0 disk.sda.out‚ 347.26171875 network.sent‚ 2.79745e+07 disk.sda2.out‚0 disk.sda.in‚ 0.0234375 cpu‚25.368636652785717לp“ˆ)‚(:ˆ ÔÞæÔ²€éÕ gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦O‰ÊE‰)‚(:º ÔÞæÔ¿¿«Ø disk.rbd1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 953.08984375) proc.memory.percent‚17.55349636077881# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda.out‚ 347.7734375 disk.sda1.out‚0 disk.sda3.in‚ 0.0234375 disk.rbd0.in‚0$ disk.nvme0n1.out‚40215.892578125! disk.dm-0.out‚40215.892578125 disk.dm-1.out‚ 347.7734375# proc.memory.rssMB‚ 2875.96484375) disk./.usagePercent‚74.52579596428905 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 disk.rbd3.out‚0% disk./.usageGB‚1110.4882164001465 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 347.7734375 network.sent‚ 2.808355e+07! disk.rbd2.out‚309.01611328125& disk.nvme0c0n1.out‚40215.892578125 disk.sda2.out‚0 disk.rbd1.out‚ 4.599609375 network.recv‚ 3.233547e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda.in‚ 0.0234375$ memory_percent‚94.18280124664307 cpu‚25.368127185366877 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1560.38671875 disk.dm-1.in‚ 0.0234375ûÿG2Š)- _wandbruntime‚8886  _runtime‚8886z1Ê‹)Ä _runtime‚8886.944581886 _step‚935 ' lr-AdamW/pg1‚1.8277034463088476e-06 ' lr-AdamW/pg2‚1.8277034463088476e-06 trainer/global_step‚11399 % _timestamp‚1.7884567953901713e+09à¢ÖÏŒ)É ' lr-AdamW/pg1‚1.8277034463088476e-06 ' lr-AdamW/pg2‚1.8277034463088476e-06 trainer/global_step‚11399 % _timestamp‚1.7884567953901713e+09 _runtime‚8886.944581886 _step‚935§>-2)- _wandbruntime‚8886  _runtime‚8886tê'Ž)äO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884567953909123e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692542464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10019443283456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673338 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530587648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54521633 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794110 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734334210387968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215311975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734334024921088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54522080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151330750 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291937 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663324163679744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165917000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10019604810240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944syèD)éO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.lÙn.†«&arge_pool.allocated‚645934350441472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672876530587648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54521633 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794110 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734334210387968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215311975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734334024921088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54522080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151330750 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291937 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663324163679744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165917000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚645934333402112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10019604810240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884567953909123e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692542464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10019443283456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673338 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560¨/¾ê2)- _wandbruntime‚8887  _runtime‚8887¸>€Íô&‘)îM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151343371 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396530061312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703067648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396717612032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151342933 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020297686016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215330455 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10020461296640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168346112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028688896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96816836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8887.626835961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663380534721536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54526533+û75ù&’)óM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553168346112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚672933028688896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96816836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8887.626835961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663380534721536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54526533 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151343371 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734396530061312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703067648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396717612032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151342933 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10020297686016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215330455 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10020461296640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944©[j' 2“)- _wandbruntime‚8887  _runtime‚8887±ZkZ”)Ô /loss/FRt3/assignment_loss‚2.893473429029638 &loss/total_loss‚42.525078837276936 _runtime‚8887.628248973 _step‚938 epoch‚9 1loss/FBt4/assignment_loss‚0.19572990258327788 /loss/FRt2/detection_loss‚1.7863133989591549 /loss/FRt3/detection_loss‚2.0494543834166272 .loss/FRt4/detection_loss‚3.110782769918442 1loss/SRqqt2/detection_loss‚1.0920755010037149 0loss/FRt1/assignment_loss‚2.6157235750337926 1loss/FBt1/assignment_loss‚0.20264260188014305 # _timestamp‚1.78845679607364e+09 2loss/SRqqt1/assignment_loss‚1.3880704701310251 /loss/FRt1/detection_loss‚2.4013784375132583 /loss/FBt3/detection_loss‚3.0686094562212625 /loss/FRt4/assignment_loss‚2.932664403915405 1loss/FBt2/assignment_loss‚0.06534850054893959 .loss/FBt2/detection_loss‚0.675814443087123 .loss/FBt4/detection_loss‚2.365013043085734 trainer/global_step‚11399 2loss/SRqqt3/assignment_loss‚1.1973884046168588 /loss/FBt3/assignment_loss‚0.395057790601489 1loss/SRqqt1/detection_loss‚1.9677683185876869 0loss/FRt2/assignment_loss‚1.5193760027276708 2loss/SRqqt2/assignment_loss‚1.2143384603417067 2loss/SRqqt4/assignment_loss‚2.1919533498585224 0loss/SRqqt3/detection_loss‚1.802913125331809 1loss/SRqqt4/detection_loss‚2.3682763539254665 .loss/FBt1/detection_loss‚3.024912714958191•¼¦_•)Ù /loss/FRt4/assignment_loss‚2.932664403915405 1loss/FBt2/assignment_loss‚0.06534850054893959 .loss/FBt2/detection_loss‚0.675814443087123 .loss/FBt4/detection_loss‚2.365013043085734 trainer/global_step‚11399 2loss/SRqqt3/assignment_loss‚1.1973884046168588 /loss/FBt3/assignment_loss‚0.395057790601489 1loss/SRqqt1/detection_loss‚1.9677683185876869 0loss/FRt2/assignment_loss‚1.5193760027276708 2loss/SRqqt2/assignment_loss‚1.2143384603417067 2loss/SRqqt4/assignment_loss‚2.1919533498585224 0loss/SRqqt3/detection_loss‚1.802913125331809 1loss/SRqqt4/detection_loss‚2.3682763539254665 .loss/FBt1/detection_loss‚3.024912714958191 /loss/FRt3/assignment_loss‚2.893473429029638 &loss/total_loss‚42.525078837276936 _runtime‚8887.628248973 _step‚938 epoch‚9 1loss/FBt4/assignment_loss‚0.19572990258327788 /loss/FRt2/detection_loss‚1.7863133989591549 /loss/FRt3/detection_loss‚2.0494543834166272 .loss/FRt4/detection_loss‚3.110782769918442 1loss/SRqqt2/detection_loss‚1.0920755010037149 0loss/FRt1/assignment_loss‚2.6157235750337926 1loss/FBt1/assignment_loss‚0.20264260188014305 # _timestamp‚1.78845679607364e+09 2loss/SRqqt1/assignment_loss‚1.3880704701310251 /loss/FRt1/detection_loss‚2.4013784375132583 /loss/FBt3/detection_loss‚3.0686094562212625ªä[ç–)‚(:ˆ ãÞæÔ"ñg€á€ÐÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.332' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.8664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇhGG—)‚(:¼ ãÞæÔ¼ŽÕØ) proc.memory.percent‚17.55349636077881$ disk./.usageGB‚1110.488338470459 disk.rbd0.in‚0# disk.dm-0.out‚40216.65966796875 cpu‚25.36756232109307& disk.nvme0n1.out‚40216.65966796875 disk.sda2.in‚0$ memory_percent‚94.18563842773438# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.dm-1.out‚ 348.26953125 disk.sda3.in‚ 0.0234375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 network.sent‚ 2.8088701e+07 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 348.26953125# proc.memory.availableMB‚952.625) disk./.usagePercent‚74.52580415653013 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.0234375 network.recv‚ 3.237435e+06 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚40216.65966796875! disk.nvme0n1.in‚ 1560.38671875 disk.sda.out‚ 348.26953125 disk.sda1.out‚0 disk.rbd2.out‚309.1689453125ü ]ž˜)‚(:“ òÞæÔ®ŽºÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚105.401' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.080199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ºß…F™)‚(:» òÞæÔ«¹ÜØ) disk./.usagePercent‚74.45648627652156$ disk./.usageGB‚1109.455451965332 disk.sda3.in‚ 0.0234375# disk.dm-0.out‚40477.74462890625' proc.memory.availableMB‚ 952.7421875 cpu‚25.367056942851328# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881 network.sent‚ 2.8197846e+07 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚ 0.0234375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 348.609375 disk.rbd2.in‚0( disk.nvme0c0n1.out‚40477.74462890625! disk.nvme0n1.in‚ 1560.38671875& disk.nvme0n1.out‚40477.74462890625 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 network.recv‚ 3.244464e+06 disk.dm-1.in‚ 0.0234375 disk.dm-1.out‚ 348.609375 disk.rbd3.in‚0 disk.sda.out‚ 348.609375 disk.sda1.out‚0 disk.rbd2.out‚309.1845703125$ memory_percent‚94.18492317199707°âKø2š)- _wandbruntime‚8922  _runtime‚8922mÖXÊ›)Ä ' lr-AdamW/pg1‚1.5734999978470176e-06 ' lr-AdamW/pg2‚1.5734999978470176e-06 trainer/global_step‚11449 % _timestamp‚1.7884568311478527e+09 _runtime‚8922.702248421 _step‚939øx.mÏœ)É trainer/global_step‚11449 % _timestamp‚1.7884568311478527e+09 _runtime‚8922.702248421 _step‚939 ' lr-AdamW/pg1‚1.5734999978470176e-06 ' lr-AdamW/pg2‚1.5734999978470176e-06«K#!â2)- _wandbruntime‚8922  _runtime‚89224³j}é'ž)ãO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216217989 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151965338 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666141833625088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557361133518 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217989 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382835787 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166617800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168335872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97217068 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54748268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9558015571274 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9592678348288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚8922.703162864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166617798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658237572540746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97217070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10062363059200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063630 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666141815622144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚675735168335872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54747826 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788456831148609e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0šgî'Ÿ)èO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216238027 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151964894 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216238027 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9593334710784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚675734493970432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737459671241216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚675735168335872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54747826 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 $ _timestamp‚1.788456831148609e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216217989 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151965338 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666141833625088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557361133518 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217989 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382835787 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166617800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168335872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97217068 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54748268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9558015571274 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9592678348288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚8922.703162864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166617798 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658237572540746 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97217070 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10062363059200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063630 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666141815622144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256507 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666437120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063214042624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 _step‚941 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521983301120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚8923.378508884 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497776128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340815872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151977887 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063383944704 trainer/global_step‚11449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9594154151936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9558177568482 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166631814 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMon,aOµùitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54752212 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151977448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9558177568482 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166631814 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54752212 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151977448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256507 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666437120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10063214042624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 _step‚941 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521983301120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚8923.378508884 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9593497776128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97225238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340815872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151977887 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10063383944704 trainer/global_step‚11449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414­«É%¶2£)- _wandbruntime‚8923  _runtime‚8923™¸`N\¤)Ö 1loss/SRqqt4/detection_loss‚2.3668353074583512 % _timestamp‚1.7884568318253446e+09 _step‚942 1loss/SRqqt2/detection_loss‚1.0341289701021237 .loss/FBt1/detection_loss‚1.972960737016466 /loss/FBt3/detection_loss‚2.4754349887371063 _runtime‚8923.379936403 epoch‚9 /loss/FRt1/assignment_loss‚2.439829681813717 2loss/SRqqt4/assignment_loss‚1.4618754358641033 1loss/FBt3/assignment_loss‚0.12121462593779028 1loss/SRqqt3/detection_loss‚1.5156849387288094 /loss/FRt4/assignment_loss‚3.459693209691481 2loss/SRqqt2/assignment_loss‚0.8872810485367866 2loss/SRqqt3/assignment_loss‚1.3276605133712291 1loss/SRqqt1/detection_loss‚1.7515612595968626 %loss/total_loss‚40.27138300311731 2loss/SRqqt1/assignment_loss‚1.1096217225964542 1loss/FBt4/assignment_loss‚0.31175202153099235 .loss/FRt4/detection_loss‚3.311268391934308 0loss/FBt2/assignment_loss‚0.0825757223561045 trainer/global_step‚11449 /loss/FRt3/assignment_loss‚3.019260409745303 /loss/FRt2/detection_loss‚1.5615141424867842 .loss/FRt3/detection_loss‚1.891375427625396 /loss/FBt4/detection_loss‚3.2886374592781067 0loss/FRt2/assignment_loss‚1.4707845570864502 0loss/FBt1/assignment_loss‚0.3839095330202124 /loss/FBt2/detection_loss‚0.6131987844808776 /loss/FRt1/detection_loss‚2.4133241141214965g¤€a¥)Û 1loss/SRqqt2/detection_loss‚1.0341289701021237 .loss/FBt1/detection_loss‚1.972960737016466 /loss/FBt3/detection_loss‚2.4754349887371063 _runtime‚8923.379936403 epoch‚9 /loss/FRt1/assignment_loss‚2.439829681813717 2loss/SRqqt4/assignment_loss‚1.4618754358641033 1loss/FBt3/assignment_loss‚0.12121462593779028 1loss/SRqqt3/detection_loss‚1.5156849387288094 /loss/FRt4/assignment_loss‚3.459693209691481 2loss/SRqqt2/assignment_loss‚0.8872810485367866 2loss/SRqqt3/assignment_loss‚1.3276605133712291 1loss/SRqqt1/detection_loss‚1.7515612595968626 %loss/total_loss‚40.27138300311731 2loss/SRqqt1/assignment_loss‚1.1096217225964542 1loss/FBt4/assignment_loss‚0.31175202153099235 .loss/FRt4/detection_loss‚3.311268391934308 0loss/FBt2/assignment_loss‚0.0825757223561045 trainer/global_step‚11449 /loss/FRt3/assignment_loss‚3.019260409745303 /loss/FRt2/detection_loss‚1.5615141424867842 .loss/FRt3/detection_loss‚1.891375427625396 /loss/FBt4/detection_loss‚3.2886374592781067 0loss/FRt2/assignment_loss‚1.4707845570864502 0loss/FBt1/assignment_loss‚0.3839095330202124 /loss/FBt2/detection_loss‚0.6131987844808776 /loss/FRt1/detection_loss‚2.4133241141214965 1loss/SRqqt4/detection_loss‚2.3668353074583512 % _timestamp‚1.7884568318253446e+09 _step‚942®ø‰y$ž¦)‚(:“ ßæÔØ’ïÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.012' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.602400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£{J§)‚(:¿ ßæÔøå¼Ù( disk.nvme0c0n1.out‚40482.21728515625 disk.rbd3.out‚0 network.sent‚ 2.8203506e+07 disk.sda.out‚ 349.1484375 disk.sda3.in‚ 0.0234375 disk.sda1.in‚0 network.recv‚ 3.248681e+06% disk./.usageGB‚1109.4553565979004! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 349.1484375( proc.memory.availableMB‚ 952.76171875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 disk.rbd1.out‚ 4.599609375) proc.memory.percent‚17.55349636077881& disk.nvme0n1.out‚40482.21728515625 disk.rbd3.in‚0$ memory_percent‚94.18480396270752 disk.drbd1777.out‚0# disk.dm-0.out‚40482.21728515625# proc.memory.rssMB‚ 2875.96484375 disk.sda1.out‚0 disk.rbd2.out‚310.8212890625 disk.dm-1.in‚ 0.0234375 disk.sda.in‚ 0.0234375 cpu‚25.366521113636683( disk./.usagePercent‚74.4564798763332 disk.rbd2.in‚0 disk.dm-1.out‚ 349.1484375 proc.cpu.threads‚17 {(H“¨)‚(:ˆ ßæÔÿÆïÕ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.529' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û0ÙìK©)‚(:À ßæÔõ€íØ disk.drbd1777.in‚0 disk.sda3.out‚ 349.51953125$ memory_percent‚94.18566226959229( proc.memory.availableMB‚ 952.62109375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.0234375' disk.nvme0c0n1.out‚40482.5478515625! disk.nvme0n1.in‚ 1560.38671875 disk.rbd1.in‚0 disk.sda3.in‚ 0.0234375 cpu‚25.365995179338395 network.sent‚ 2.8312726e+07 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.0234375 disk.rbd0.out‚0% disk.nvme0n1.out‚40482.5478515625 disk.sda.out‚ 349.51953125# proc.memory.rssMB‚ 2875.96484375# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17% disk./.usageGB‚1109.4555702209473 disk.rbd2.out‚310.8369140625" disk.dm-0.out‚40482.5478515625 disk.dm-1.out‚ 349.51953125 disk.sda1.out‚0 network.recv‚ 3.256066e+06 disk.rbd2.in‚0 disk.rbd1.out‚ 4.599609375 disk.rbd3.in‚0 disk.sda1.in‚0) disk./.usagePercent‚74.45649421275512 òlžª)‚(:“ ŸßæÔÖ‚ÄÕ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.232' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.846399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÑw<«)‚(:± ŸßæÔ›ú‰Ø# proc.memory.rssMB‚ 2875.96484375 disk.rbd1.out‚ 4.599609375 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-1.out‚350.125$ memory_percent‚94.18420791625977( disk./.usagePercent‚74.4564954927928 disk.rbd2.out‚310.8994140625 disk.sda.in‚ 0.0234375 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚40482.97509765625 disk.sda.out‚350.125 disk.sda1.in‚0! disk.nvme0n1.in‚ 1560.38671875& proc.memory.availableMB‚ 952.859375 cpu‚25.365444112233696 network.recv‚ 3.260017e+06 disk.rbd3.in‚0 disk.sda3.in‚ 0.0234375# disk.dm-0.out‚40482.97509765625 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1109.4555892944336 disk.rbd1.in‚0 disk.dm-1.in‚ 0.0234375 disk.rbd3.out‚0 disk.sda3.out‚350.125 disk.rbd0.out‚0 disk.dm-0.in‚ 1560.38671875) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 network.sent‚ 2.8317852e+07& disk.nvme0n1.out‚40482.97509765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0!7®Ž2¬)- _wandbruntime‚8958  _runtime‚8958@/]Ê­)Ä _runtime‚8958.704331882 _step‚943 ' lr-AdamW/pg1‚1.3381709278121147e-06 ' lr-AdamW/pg2‚1.3381709278121147e-06 trainer/global_step‚11499 % _timestamp‚1.7884568671498704e+09ð,«âÏ®)É ' lr-AdamW/pg1‚1.3381709278121147e-06 ' lr-AdamW/pg2‚1.3381709278121147e-06 trainer/global_step‚11499 % _timestamp‚1.7884568671498704e+09 _runtime‚8958.704331882 _step‚943¯U@7Ä2¯)- _wandbruntime‚8958  _runtime‚8958†Œ(½é'°)ãO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54975355 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806084096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152600389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462639 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97625468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10105456740352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚ 8958.70521269 _step‚944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740585112144384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167318600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105626655744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152600825 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740585305999872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97625470 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217164079 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144041 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664154112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884568671506166e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608©á±" ±)èO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678593806084096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152600389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384462639 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97625468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10105456740352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚ 8958.70521269 _step‚944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740585112144384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167318600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105626655744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152600825 ^JDeviceStatsMonitor.ª±2Ìon_train_batch_start/inactive_split_bytes.all.allocated‚740585305999872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97625470 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217164079 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217144041 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664154112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884568671506166e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54975355 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167318598 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11499°zc¤02²)- _wandbruntime‚8959  _runtime‚8959Ÿ¨ãò&³)ìM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10106480039424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152613577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788456867809838e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217162495 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106308040192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217182559 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647810121216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9635139957760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661078663496562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 8959.36487874 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651479667628032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678650304185344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661079334973886 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54979485 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162495 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384495109 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152613121 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162495 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384495109 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152613121 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152613577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788456867809838e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217162495 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106308040192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217182559 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97633638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647810121216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9635139957760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661078663496562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 8959.36487874 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651479667628032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678650304185344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661079334973886±¢Î*2µ)- _wandbruntime‚8959  _runtime‚8959púºÎU¶)Ï 0loss/FBt3/assignment_loss‚0.3846472024710642 /loss/FRt3/detection_loss‚2.0708362665914355 trainer/global_step‚11499 1loss/SRqqt2/assignment_loss‚1.022943014773773 .loss/FBt3/detection_loss‚2.420884715186225 _runtime‚8959.366329351 /loss/FRt2/assignment_loss‚1.140138419561608 /loss/FRt1/detection_loss‚2.3180513194827146 1loss/SRqqt1/detection_loss‚1.9115774803187535 0loss/SRqqt4/detection_loss‚2.387840235233307 .loss/FBt4/detection_loss‚2.877039381435939 %loss/total_loss‚40.94709057073692 1loss/SRqqt1/assignment_loss‚1.184716235279389 2loss/SRqqt4/assignment_loss‚1.5968862469707217 /loss/FRt2/detection_loss‚1.5772451193959907 /loss/FRt3/assignment_loss‚3.029758578255063 0loss/FRt4/assignment_loss‚3.3232668457609233 0loss/SRqqt3/assignment_loss‚1.31778739190296 .loss/FRt4/detection_loss‚2.699381026354703 1loss/SRqqt2/detection_loss‚1.0517290418501943 .loss/FBt1/detection_loss‚2.376712679862976 1loss/FBt1/assignment_loss‚0.25850427164862466 0loss/FBt2/assignment_loss‚0.0828007076456561 /loss/FBt4/assignment_loss‚0.412071487999242 epoch‚9 _step‚946 /loss/FRt1/assignment_loss‚2.986529060574465 1loss/SRqqt3/detection_loss‚2.0803811992316144 0loss/FBt2/detection_loss‚0.43536264294957666 % _timestamp‚1.7884568678120193e+09“‘²HZ·)Ô 0loss/FBt3/assignment_loss‚0.3846472024710642 /loss/FRt3/detection_loss‚2.0708362665914355 trainer/global_step‚11499 1loss/SRqqt2/assignment_loss‚1.022943014773773 .loss/FBt3/detection_loss‚2.420884715186225 _runtime‚8959.366329351 /loss/FRt2/assignment_loss‚1.140138419561608 /loss/FRt1/detection_loss‚2.3180513194827146 1loss/SRqqt1/detection_loss‚1.9115774803187535 0loss/SRqqt4/detection_loss‚2.387840235233307 .loss/FBt4/detection_loss‚2.877039381435939 %loss/total_loss‚40.94709057073692 1loss/SRqqt1/assignment_loss‚1.184716235279389 2loss/SRqqt4/assignment_loss‚1.5968862469707217 /loss/FRt2/detection_loss‚1.5772451193959907 /loss/FRt3/assignment_loss‚3.029758578255063 0loss/FRt4/assignment_loss‚3.3232668457609233 0loss/SRqqt3/assignment_loss‚1.31778739190296 .loss/FRt4/detection_loss‚2.699381026354703 1loss/SRqqt2/detection_loss‚1.0517290418501943 .loss/FBt1/detection_loss‚2.376712679862976 1loss/FBt1/assignment_loss‚0.25850427164862466 0loss/FBt2/assignment_loss‚0.0828007076456561 /loss/FBt4/assignment_loss‚0.412071487999242 epoch‚9 _step‚946 /loss/FRt1/assignment_loss‚2.986529060574465 1loss/SRqqt3/detection_loss‚2.0803811992316144 0loss/FBt2/detection_loss‚0.43536264294957666 % _timestamp‚1.7884568678120193e+09²-üTÄ“¸)‚(:ˆ ®ßæÔóã‚× gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gŠã‘G¹)‚(:¼ ®ßæÔ„¤ÍØ disk.dm-1.out‚ 350.484375 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.0234375 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚40483.32666015625 disk.dm-0.in‚ 1560.38671875 network.sent‚ 2.8323992e+07% disk./.usageGB‚1109.4553146362305# disk.dm-0.out‚40483.32666015625# proc.memory.rssMB‚ 2875.96484375) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.45647706025032 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚310.9150390625 disk.sda2.in‚0' proc.memory.availableMB‚ 952.2890625 disk.rbd1.out‚ 4.599609375 disk.rbd3.out‚0 disk.sda.in‚ 0.0234375 disk.sda3.in‚ 0.0234375 disk.sda3.out‚ 350.484375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1560.38671875 network.recv‚ 3.264371e+06 disk.sda.out‚ 350.484375 disk.drbd1777.out‚0 disk.sda2.out‚0 cpu‚25.364942812107643( disk.nvme0c0n1.out‚40483.32666015625$ memory_percent‚94.18768882751465 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1560.38671875õ Ô]œº)‚(:‘ ½ßæÔ¶§ÝÕ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.4' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.880000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gœ†J»)‚(:¿ ½ßæÔ³ëó× disk.sda3.in‚ 0.0234375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 cpu‚25.3644226346033 disk.sda.out‚ 351.1640625 disk.drbd1777.in‚0% disk./.usageGB‚1109.4552764892578 disk.dm-1.in‚ 0.0234375 disk.sda1.out‚0& disk.nvme0n1.out‚40484.54150390625$ memory_percent‚94.18704509735107 disk.sda3.out‚ 351.1640625 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875! disk.nvme0n1.in‚ 1560.38671875( proc.memory.availableMB‚ 952.39453125# proc.memory.rssMB‚ 2875.96484375 disk.rbd1.out‚ 4.599609375 network.sent‚ 2.8432284e+07 disk.sda2.out‚0 disk.dm-1.out‚ 351.1640625 disk.sda.in‚ 0.0234375 disk.sda1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚311.00537109375# disk.dm-0.out‚40484.54150390625 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.recv‚ 3.271429e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚40484.54150390625) proc.memory.percent‚17.55349636077881) disk./.usagePercent‚74.45647450017499QOí$¼)- _wandbruntime‚8994  _ê^xvruntime‚8994ú'½|Ƚ) trainer/global_step‚11549 % _timestamp‚1.7884569027576897e+09 _runtime‚8994.312060445 _step‚947 & lr-AdamW/pg1‚1.121765278429615e-06 & lr-AdamW/pg2‚1.121765278429615e-06Lfù¢Í¾)Ç & lr-AdamW/pg1‚1.121765278429615e-06 & lr-AdamW/pg2‚1.121765278429615e-06 trainer/global_step‚11549 % _timestamp‚1.7884569027576897e+09 _runtime‚8994.312060445 _step‚947³)¯¶2¿)- _wandbruntime‚8994  _runtime‚8994¼kµê'À)äO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681451769466880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168019400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10148430592000 _runtime‚8994.313006226 _step‚948 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654169952986112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚11549 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168019400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153235155 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884569027584383e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649959936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168019398 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654169952986112 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚11549 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168019400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153235155 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884569027584383e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649959936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168019398 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944´ZB'2Â)- _wandbruntime‚8994  _runtime‚8994=¾`ô&Ã)îM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884569034243367e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247298 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773128330752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9675469387776 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218088547 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681509616312320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168033414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55205425 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168033416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98042038 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8994.979211476 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153247461 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773317978624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941933568 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30-ç’‰íÄ)óM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218108611 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98042036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10149451013632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚743773128330752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654224857116672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9675469387776 J;DeviceStatsMonitor.on_train_batchˆ‘ÑÆ _end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121961 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218088547 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚681509616312320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168033414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55205425 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚671833472545792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168033416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98042038 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733623866964992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8994.979211476 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚671833490548736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153247461 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743773317978624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941933568 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884569034243367e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9639848247298 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884569382804375e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55426315 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153869023 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191584698368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407215104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666595698142970 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153868583 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081580544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635765760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168720198 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219016183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635765760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168720198 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219016183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884569382804375e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55426315 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153869023 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191584698368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚952 R>DeviceStatsMonitor.on_train_batch_start/active_byte§ïs.all.freed‚684310407215104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚674594771449344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666595698142970 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153868583 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚684311081580544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880¸J8žÒ)‚(:“ êßæÔÁ‘ÍÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.701' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.740199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨¡ÛIÓ)‚(:¾ êßæÔÒ×ü×$ disk./.usageGB‚1109.455551147461 disk.dm-0.in‚ 1560.38671875 disk.sda2.out‚0 disk.dm-1.in‚ 0.0234375 disk.sda.in‚ 0.0234375 disk.sda3.in‚ 0.0234375' disk.nvme0c0n1.out‚40485.9658203125) disk./.usagePercent‚74.45649293271745 disk.rbd1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚312.65771484375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 352.77734375" disk.dm-0.out‚40485.9658203125 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 352.77734375 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.8552851e+07 disk.dm-1.out‚ 352.77734375! disk.nvme0n1.in‚ 1560.38671875# memory_percent‚94.1890001296997( proc.memory.availableMB‚ 952.07421875 network.recv‚ 3.28695e+06 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2875.96484375 proc.cpu.threads‚17 disk.drbd1777.in‚0# disk.nvme0c0n1.in‚ 1560.38671875% disk.nvme0n1.out‚40485.9658203125) proc.memory.percent‚17.55349636077881 cpu‚25.362832167798175ðK†ã2Ô)- _wandbruntime‚9030  _runtime‚9030ÿ?>$ô&Õ)îM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98450438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153881214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884569389429884e+09 _step‚953 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219014599 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10192448771584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10192280966656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684368254060544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666651418708702 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9680683586706 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55430778 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254060544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153881665 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024670 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚9030.498145968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025(2ù&Ö)óM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55430778 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254060544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153881665 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9681338024670 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚9030.498145968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98450438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153881214 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884569389429884e+09 _step‚953 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219014599 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10192448771584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10192280966656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684368254060544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666651418708702 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9680683586706 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813¹„<©2×)- _wandbruntime‚9030  _runtime‚9030ÇâÀ/[Ø)Õ $ _timestamp‚1.788456938944957e+09 0loss/FRt2/assignment_loss‚1.6040246109573209 2loss/SRqqt1/assignment_loss‚1.3503337783600997 /loss/FBt2/detection_loss‚0.6169965106042206 epoch‚9 _runtime‚9030.499294712 0loss/FRt1/assignment_loss‚2.3398111164569855 /loss/FRt3/assignment_loss‚2.740961553020911 1loss/FBt3/assignment_loss‚0.31435938500281835 1loss/SRqqt3/assignment_loss‚1.396041923435405 0loss/FBt4/assignment_loss‚0.2774001131620025 trainer/global_step‚11599 2loss/SRqqt4/assignment_loss‚1.6081299441747177 0loss/FBt1/assignment_loss‚0.0630184181633611 /loss/FRt2/detection_loss‚1.7973612925341846 1loss/SRqqt2/detection_loss‚0.9065268396108008 0loss/SRqqt3/detection_loss‚1.775561525752502 &loss/total_loss‚40.777687155200624 _step‚954 /loss/FRt4/assignment_loss‚3.197617435455322 1loss/FBt2/assignment_loss‚0.12949697703720095 /loss/FRt1/detection_loss‚2.3732650869271974 0loss/SRqqt1/detection_loss‚1.708538266243758 /loss/FBt3/detection_loss‚2.7437247236569724 0loss/SRqqt4/detection_loss‚2.192017512443738 /loss/FBt1/detection_loss‚2.9032671517795987 2loss/SRqqt2/assignment_loss‚1.0176748961710662 /loss/FRt3/detection_loss‚2.0689852413806062 /loss/FRt4/detection_loss‚2.9237543841203055 .loss/FBt4/detection_loss‚2.728818468749523t©¹õ`Ù)Ú $ _timestamp‚1.788456938944957e+09 0loss/FRt2/assignment_loss‚1.6040246109573209 2loss/SRqqt1/assignment_loss‚1.3503337783600997 /loss/FBt2/detection_loss‚0.6169965106042206 epoch‚9 _runtime‚9030.499294712 0loss/FRt1/assignment_loss‚2.3398111164569855 /loss/FRt3/assignment_loss‚2.740961553020911 1loss/FBt3/assignment_loss‚0.31435938500281835 1loss/SRqqt3/assignment_loss‚1.396041923435405 0loss/FBt4/assignment_loss‚0.2774001131620025 trainer/global_step‚11599 2loss/SRqqt4/assignment_loss‚1.6081299441747177 0loss/FBt1/assignment_loss‚0.0630184181633611 /loss/FRt2/detection_loss‚1.7973612925341846 1loss/SRqqt2/detection_loss‚0.9065268396108008 0loss/SRqqt3/detection_loss‚1.775561525752502 &loss/total_loss‚40.777687155200624 _step‚954 /loss/FRt4/assignment_loss‚3.197617435455322 1loss/FBt2/assignment_loss‚0.12949697703720095 /loss/FRt1/detection_loss‚2.3732650869271974 0loss/SRqqt1/detection_loss‚1.708538266243758 /loss/FBt3/detection_loss‚2.7437247236569724 0loss/SRqqt4/detection_loss‚2.192017512443738 /loss/FBt1/detection_loss‚2.9032671517795987 2loss/SRqqt2/assignment_loss‚1.0176748961710662 /loss/FRt3/detection_loss‚2.0689852413806062 /loss/FRt4/detection_loss‚2.9237543841203055 .loss/FBt4/detection_loss‚2.728818468749523ºZƒ7’Ú)‚(:‡ ùßæÔÞ¾ÂÕ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿_á‰LÛ)‚(:Á ùßæÔÖäÒ×# proc.memory.rssMB‚ 2875.96484375( disk.nvme0c0n1.out‚40490.10205078125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd2.out‚ 312.826171875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1560.38671875 cpu‚25.36231675463453) disk./.usagePercent‚74.44519711228416& disk.nvme0n1.out‚40490.10205078125$ memory_percent‚94.18838024139404 network.sent‚ 2.8649827e+07 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.0234375% disk./.usageGB‚1109.2872352600098 disk.dm-0.in‚ 1560.38671875# disk.dm-0.out‚40490.10205078125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.0234375 disk.sda3.out‚ 353.58984375) proc.memory.percent‚17.55349636077881 disk.rbd1.out‚ 4.599609375 network.recv‚ 3.293084e+06# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-1.out‚ 353.58984375 disk.sda.out‚ 353.58984375( proc.memory.availableMB‚ 952.17578125p \8Ü)‚(:… ˆàæÔ“þŸÕ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òRÝ´?Ý)‚(:´ ˆàæÔ©óÕØ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 354.734375# proc.memory.rssMB‚ 2875.96484375 cpu‚25.36176775983538 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.in‚0# memory_percent‚94.1878080368042) proc.memory.percent‚17.55349636077881 proc.cpu.threads‚17 disk.sda.in‚ 0.0234375 disk.sda3.out‚ 354.734375 disk.dm-0.in‚ 1560.38671875( proc.memory.availableMB‚ 952.26953125 disk.rbd3.out‚0 disk.sda.out‚ 354.734375 disk.sda3.in‚ 0.0234375 disk.rbd2.out‚ 312.826171875! disk.dm-0.out‚40490.451171875 network.recv‚ 3.299747e+06% disk./.usageGB‚1109.2872352600098 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375 disk.rbd3.in‚0& disk.nvme0c0n1.out‚40490.451171875$ disk.nvme0n1.out‚40490.451171875 disk.dm-1.in‚ 0.0234375 network.sent‚ 2.8748121e+07 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875) disk./.usagePercent‚74.44519711228416 disk.drbd1777.in‚0 disk.drbd1777.out‚0y),2Þ)- _wandbruntime‚9065  _runtime‚9065œKM€Çß)Á & lr-AdamW/pg1‚7.459006830627772e-07 & lr-AdamW/pg2‚7.459006830627772e-07 trainer/global_step‚11649 % _timestamp‚1.7884569742051408e+09 _runtime‚ 9065.75957866 _step‚955uŠéÌà)Æ & lr-AdamW/pg1‚7.459006830627772e-07 & lr-AdamW/pg2‚7.459006830627772e-07 trainer/global_step‚11649 % _timestamp‚1.7884569742051408e+09 _runtime‚ 9065.75957866 _step‚955»Zç(å2á)- _wandbruntime‚9065  _runtime‚9065>–=•ü â)äO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677412441394688 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154502862 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884569742143133e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219942235 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219922197 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10234549425152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044963328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98850668 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚956 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.¡Qœ‚îall.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277934080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010378 trainer/global_step‚11649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154503305 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9756621571584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219922197 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚659660383081472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719328768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚669381068533262 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9720702491150 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961649757696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚9065.768959441 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55652635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961839418880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234383704064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169421000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9721356928906 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277934080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010378 trainer/global_step‚11649 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154503305 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9756621571584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219922197 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚659660383081472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687169719328768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚669381068533262 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9720702491150 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961649757696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 _runtime‚9065.768959441 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55652635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961839418880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234383704064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169421000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096¼¶b# 2ä)- _wandbruntime‚9066  _runtime‚9066PBTô&å)îM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55656647 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169435014 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154515483 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389375665 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219960715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659715287212032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235412050432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55657107 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024163133952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154515945 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097375232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677468794433536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9757440999424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98858838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219940651 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169435014 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024352781824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687226891808768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚669437460576110 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚669436789098786 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677468776430592sþþŒÚæ)óM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169435016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739788916791296 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219940651 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235412050432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55657107 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024163133952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154515945 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097375232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677468794433536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9757440999424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98858838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219940651 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169435014 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024352781824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687226891808768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚669437460576110 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚669436789098786 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677468776430592 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.ppÖéÍeak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55656647 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 trainer/global_step‚11649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169435014 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154515483 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389375665 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219960715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659715287212032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9762192268314 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170121800 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155137502 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690028357076992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087265295872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690028357076992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087265295872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9762192268314À<=‘k2ô)- _wandbruntime‚9102  _runtime‚9102¢(²õ)íM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _runtime‚ 9102.25985005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_e‘†ãØA$nd/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220866703 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 _step‚961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391002517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚753149790441984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884570107045984e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886767 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170135814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662460493740032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220866703 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680286446375936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022583 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155150144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155150572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278336767488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426805248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220866703 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99267238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 _step‚961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391002517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚753149790441984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884570107045984e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220886767 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170135814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚680286428372992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662460493740032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220866703 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚680286446375936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022583 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155150144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155150572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10278336767488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9798426805248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 B½’þ)‚(:‡ ÓàæÔ…‰ßÕ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚92.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ÐôKÿ)‚(:À ÓàæÔÜžÀØ disk.dm-1.in‚ 0.0234375 network.recv‚ 3.324572e+06 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 358.2109375 disk.sda3.in‚ 0.0234375! disk.rbd2.out‚314.72509765625! disk.nvme0n1.in‚ 1560.38671875) proc.memory.availableMB‚ 1030.80859375% disk./.usageGB‚1109.2855949401855 disk.rbd2.in‚0 cpu‚25.35921612022747 disk.dm-1.out‚ 358.2109375 disk.rbd3.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.44508702904444 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 358.2109375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1560.38671875( disk.nvme0c0n1.out‚40493.86474609375# memory_percent‚93.7084436416626 disk.dm-0.in‚ 1560.38671875 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375& disk.nvme0n1.out‚40493.86474609375# proc.memory.rssMB‚ 2796.98828125) proc.memory.percent‚17.07146167755127 network.sent‚ 2.8981387e+07# disk.dm-0.out‚40493.86474609375 disk.sda1.out‚0–R¾ 2€*- _wandbruntime‚9137  _runtime‚9137“°´Ç*Á & lr-AdamW/pg2‚4.462195138959418e-07 trainer/global_step‚11749 $ _timestamp‚1.788457046090803e+09 _runtime‚9137.645233695 _step‚963 & lr-AdamW/pg1‚4.462195138959418e-07¡»4îÌ‚*Æ $ _timestamp‚1.788457046090803e+09 _runtime‚9137.645233695 _step‚963 & lr-AdamW/pg1‚4.462195138959418e-07 & lr-AdamW/pg2‚4.462195138959418e-07 trainer/global_step‚11749Ãâ%'A2ƒ*- _wandbruntime‚9137  _runtime‚9137+@Äׄ*äO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99667468 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794339 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚745892363760128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚9137.646155691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268078 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9838593183232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320549467136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚11749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚692886994825216 `LDeviceStatsMonitor.on_train0Äï_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56106302 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212731954688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745194 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9838593183232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99667470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212731954688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745194 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9838593183232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99667470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665150779098112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170822598 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚745892363760128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚9137.646155691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268078 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9838593183232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320549467136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚11749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844ÄÎѬr2†*- _wandbruntime‚9138  _runtime‚9138Ÿ€Iôó&‡*íM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155786852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412611072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56111214 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321253563904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9839412611072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99675636 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042894 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚9138.320115853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683104098318336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275244121600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155786415 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚692943492926464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275429575168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 trainer/global_step‚11749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068986880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392629369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788457046765316e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56110779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9840068986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99675638 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¾Ì)¶ˆ*òM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412611072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56111214 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321253563904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9839412611072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99675636 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042894 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚9138.320115853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683104098318336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚756275244121600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155786415 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚692943492926464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275429575168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 trainer/global_step‚11749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9840068986880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392629369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683104080315392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683104080315392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788457046765316e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56110779 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_traiø™ÁBn_batch_end/allocation.large_pool.freed‚ 170836614 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9840068986880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99675638 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155786852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096Å+}!É2‰*- _wandbruntime‚9138  _runtime‚9138úlŸ`Š*Ú /loss/FBt2/detection_loss‚0.4607667668802299 _step‚966 2loss/SRqqt2/assignment_loss‚0.6312636342190672 0loss/FBt2/assignment_loss‚0.1525004867192349 1loss/SRqqt4/detection_loss‚2.6341052579879762 % _timestamp‚1.7884570467668974e+09 trainer/global_step‚11749 1loss/SRqqt4/assignment_loss‚1.609706583674997 /loss/FRt4/detection_loss‚2.7387369237840176 /loss/FBt4/detection_loss‚2.3921866913636527 epoch‚9 _runtime‚9138.321219512 0loss/FRt1/assignment_loss‚2.3812363790530786 0loss/FRt3/assignment_loss‚3.0218278262350293 2loss/FBt1/assignment_loss‚0.060304876104510186 /loss/FRt2/detection_loss‚1.4890677293762564 1loss/SRqqt1/detection_loss‚1.5347591451597664 /loss/FBt3/detection_loss‚2.1038465683276835 %loss/total_loss‚39.38984645436959 /loss/FRt2/assignment_loss‚1.210036391019821 2loss/SRqqt3/assignment_loss‚1.4436520921384417 0loss/FBt3/assignment_loss‚0.2223019261772816 1loss/FBt4/assignment_loss‚0.35152998186337453 .loss/FRt3/detection_loss‚2.255465861823824 2loss/SRqqt1/assignment_loss‚1.2584092164706473 /loss/FRt1/detection_loss‚2.2813510424366186 0loss/FRt4/assignment_loss‚3.0369440093636513 1loss/SRqqt2/detection_loss‚1.0970637849532068 1loss/SRqqt3/detection_loss‚1.9483359081716072 /loss/FBt1/detection_loss‚3.0744473710656166®ïpÌe‹*ß 1loss/SRqqt4/assignment_loss‚1.609706583674997 /loss/FRt4/detection_loss‚2.7387369237840176 /loss/FBt4/detection_loss‚2.3921866913636527 epoch‚9 _runtime‚9138.321219512 0loss/FRt1/assignment_loss‚2.3812363790530786 0loss/FRt3/assignment_loss‚3.0218278262350293 2loss/FBt1/assignment_loss‚0.060304876104510186 /loss/FRt2/detection_loss‚1.4890677293762564 1loss/SRqqt1/detection_loss‚1.5347591451597664 /loss/FBt3/detection_loss‚2.1038465683276835 %loss/total_loss‚39.38984645436959 /loss/FRt2/assignment_loss‚1.210036391019821 2loss/SRqqt3/assignment_loss‚1.4436520921384417 0loss/FBt3/assignment_loss‚0.2223019261772816 1loss/FBt4/assignment_loss‚0.35152998186337453 .loss/FRt3/detection_loss‚2.255465861823824 2loss/SRqqt1/assignment_loss‚1.2584092164706473 /loss/FRt1/detection_loss‚2.2813510424366186 0loss/FRt4/assignment_loss‚3.0369440093636513 1loss/SRqqt2/detection_loss‚1.0970637849532068 1loss/SRqqt3/detection_loss‚1.9483359081716072 /loss/FBt1/detection_loss‚3.0744473710656166 /loss/FBt2/detection_loss‚0.4607667668802299 _step‚966 2loss/SRqqt2/assignment_loss‚0.6312636342190672 0loss/FBt2/assignment_loss‚0.1525004867192349 1loss/SRqqt4/detection_loss‚2.6341052579879762 % _timestamp‚1.7884570467668974e+09 trainer/global_step‚11749Ælj+G‘Œ*‚(:† âàæÔ¯ñ»Õ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÔP*‚(:Å âàæÔ›¤Ö× disk.rbd1.out‚ 4.599609375 disk.sda1.out‚0! disk.nvme0n1.in‚ 1560.38671875 disk.dm-1.out‚ 358.56640625) proc.memory.availableMB‚ 1030.77734375 network.sent‚ 2.8987592e+07 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.0234375 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚40495.37646484375# proc.memory.rssMB‚ 2796.98828125 cpu‚25.358688200617603) disk./.usagePercent‚74.44508805307458% disk./.usageGB‚1109.2856101989746 disk.sda3.out‚ 358.56640625 disk.rbd0.out‚0 disk.drbd1777.in‚0$ memory_percent‚93.70863437652588) proc.memory.percent‚17.07146167755127 disk.sda.out‚ 358.56640625 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875# disk.dm-0.out‚40495.37646484375 disk.dm-1.in‚ 0.0234375 network.recv‚ 3.328889e+06! disk.rbd2.out‚314.72509765625& disk.nvme0n1.out‚40495.37646484375 disk.sda2.out‚0 proc.cpu.threads‚17òWA“Ž*‚(:ˆ ñàæÔêþÞÕ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚151.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 YúwK*‚(:À ñàæÔžª¿Ø disk.sda3.in‚ 0.0234375 disk.rbd3.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚40622.36474609375 network.sent‚ 2.9095877e+07) disk./.usagePercent‚74.45338909737232 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.07146167755127! disk.rbd2.out‚314.80322265625 disk.dm-0.in‚ 1560.38671875) proc.memory.availableMB‚ 1030.87890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd3.in‚0% disk./.usageGB‚1109.4093017578125( disk.nvme0c0n1.out‚40622.36474609375! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.out‚0 disk.dm-1.out‚ 359.2890625 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 359.2890625$ memory_percent‚93.70801448822021 disk.sda3.out‚ 359.2890625 cpu‚25.3581524129033 proc.cpu.threads‚17# disk.dm-0.out‚40622.36474609375 disk.sda.in‚ 0.0234375# proc.memory.rssMB‚ 2796.98828125 network.recv‚ 3.336163e+06 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.0234375 disk.sda1.in‚0`¬2*- _wandbruntime‚9173  _runtime‚9173¸óhvÈ‘* & lr-AdamW/pg2‚3.250282630853395e-07 trainer/global_step‚11799 % _timestamp‚1.7884570818953142e+09 _runtime‚9173.449716439 _step‚967 & lr-AdamW/pg1‚3.250282630853395e-07E<í\Í’*Ç & lr-AdamW/pg1‚3.250282630853395e-07 & lr-AdamW/pg2‚3.250282630853395e-07 trainer/global_step‚11799 % _timestamp‚1.7884570818953142e+09 _runtime‚9173.449716439 _step‚967Ç`æ2“*- _wandbruntime‚9173  _runtime‚9173]Aç&ì'”*æO UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100075868 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222720391 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156410295 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171523398 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667895985626112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171523398 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843208509374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171523400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9880235351552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759338645670912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10363577209344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879578989056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚685865379218944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 _runtime‚9173.450607668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100075870 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695745632573440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334853 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222700353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394243791 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156410723 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 % _timestamp‚1.7884570818960452e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156410295 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171523398 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667895985626112 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171523398 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9843208509374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171523400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9880235351552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759338645670912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10363577209344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879578989056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚685865379218944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 _runtime‚9173.450607668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100075870 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695745632573440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56334853 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222700353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394243791 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156410723 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612602 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 % _timestamp‚1.7884570818960452e+09 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156423054 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚759401154336768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10364602955776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚9174.115746889 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_sùÜÒáeplit_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9844679382302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9880398416896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚969 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805053440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844024944338 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944Êà Êú&˜*ôM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156423054 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚759401154336768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10364602955776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚9174.115746889 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9844679382302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9880398416896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚969 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805053440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9844024944338 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667950889756672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30É)‹ý2™*- _wandbruntime‚9174  _runtime‚9174IOÕYš*Ó 0loss/FBt3/assignment_loss‚0.6073744632298193 _step‚970 2loss/SRqqt1/assignment_loss‚1.3386170142811948 /loss/FRt1/detection_loss‚2.4458237489064536 1loss/SRqqt1/detection_loss‚1.5086866707123558 % _timestamp‚1.7884570825625236e+09 &loss/total_loss‚44.157501202627024 0loss/FRt4/assignment_loss‚3.1413202502510766 2loss/SRqqt3/assignment_loss‚1.5725728943943977 .loss/FBt3/detection_loss‚2.415939565215792 0loss/FRt3/assignment_loss‚3.0865431889891624 2loss/SRqqt2/assignment_loss‚1.1997578481160518 .loss/FRt2/detection_loss‚2.107809189707041 0loss/FRt2/assignment_loss‚1.6686469064818488 2loss/SRqqt4/assignment_loss‚1.9430331510882224 0loss/FBt1/assignment_loss‚0.5833326100182603 trainer/global_step‚11799 _runtime‚9174.116821734 1loss/FBt2/assignment_loss‚0.09385712830015336 0loss/SRqqt4/detection_loss‚2.631093862556642 0loss/FRt1/assignment_loss‚2.8048889003694057 .loss/FRt3/detection_loss‚2.694279643148184 1loss/SRqqt3/detection_loss‚1.7584825618111568 .loss/FBt1/detection_loss‚3.323532685637474 .loss/FBt2/detection_loss‚0.535718774801741 1loss/FBt4/assignment_loss‚0.24306112250778825 .loss/FRt4/detection_loss‚3.365034967660904 /loss/SRqqt2/detection_loss‚0.86791519401595 .loss/FBt4/detection_loss‚2.220178860425949 epoch‚9W\é^›*Ø /loss/SRqqt2/detection_loss‚0.86791519401595 .loss/FBt4/detection_loss‚2.220178860425949 epoch‚9 0loss/FBt3/assignment_loss‚0.6073744632298193 _step‚970 2loss/SRqqt1/assignment_loss‚1.3386170142811948 /loss/FRt1/detection_loss‚2.4458237489064536 1loss/SRqqt1/detection_loss‚1.5086866707123558 % _timestamp‚1.7884570825625236e+09 &loss/total_loss‚44.157501202627024 0loss/FRt4/assignment_loss‚3.1413202502510766 2loss/SRqqt3/assignment_loss‚1.5725728943943977 .loss/FBt3/detection_loss‚2.415939565215792 0loss/FRt3/assignment_loss‚3.0865431889891624 2loss/SRqqt2/assignment_loss‚1.1997578481160518 .loss/FRt2/detection_loss‚2.107809189707041 0loss/FRt2/assignment_loss‚1.6686469064818488 2loss/SRqqt4/assignment_loss‚1.9430331510882224 0loss/FBt1/assignment_loss‚0.5833326100182603 trainer/global_step‚11799 _runtime‚9174.116821734 1loss/FBt2/assignment_loss‚0.09385712830015336 0loss/SRqqt4/detection_loss‚2.631093862556642 0loss/FRt1/assignment_loss‚2.8048889003694057 .loss/FRt3/detection_loss‚2.694279643148184 1loss/SRqqt3/detection_loss‚1.7584825618111568 .loss/FBt1/detection_loss‚3.323532685637474 .loss/FBt2/detection_loss‚0.535718774801741 1loss/FBt4/assignment_loss‚0.24306112250778825 .loss/FRt4/detection_loss‚3.365034967660904ÊžáÈ!žœ*‚(:“ €áæÔýœÌÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.927' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.185399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LcË¿C*‚(:¸ €áæÔÀ•µÙ network.sent‚ 2.9102128e+07 network.recv‚ 3.341087e+06$ disk.nvme0n1.out‚40749.427734375# proc.memory.rssMB‚ 2796.98828125 cpu‚25.35761793606245 proc.cpu.threads‚17 disk.sda.out‚ 359.6640625 disk.sda1.in‚0 disk.sda1.out‚0! disk.rbd2.out‚316.37744140625& disk.nvme0c0n1.out‚40749.427734375 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.599609375$ memory_percent‚93.70946884155273 disk.rbd3.in‚0 disk.sda3.in‚ 0.0234375 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1560.38671875 disk.dm-1.in‚ 0.0234375 disk.dm-1.out‚ 359.6640625$ disk./.usageGB‚1109.533920288086 disk.sda3.out‚ 359.6640625 disk.rbd0.in‚0! disk.dm-0.out‚40749.427734375) disk./.usagePercent‚74.46175235150089! disk.nvme0n1.in‚ 1560.38671875 disk.sda.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 1030.640625) proc.memory.percent‚17.07146167755127 disk.rbd0.out‚0ÀºÆ“ž*‚(:ˆ áæÔ•†âÕ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…q§ð<Ÿ*‚(:± áæÔ¦ÈØ) disk./.usagePercent‚74.46175363153856 disk.dm-1.in‚ 0.0234375 network.sent‚ 2.9210272e+07 network.recv‚ 3.34764e+06 disk.dm-1.out‚ 360.2734375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0234375 disk.sda3.out‚ 360.2734375 disk.rbd0.out‚0% disk./.usageGB‚1109.5339393615723! disk.rbd2.out‚316.45556640625! disk.nvme0n1.out‚ 40749.921875# disk.nvme0c0n1.out‚ 40749.921875! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.sda.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-0.in‚ 1560.38671875( proc.memory.availableMB‚ 1030.9453125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375 disk.drbd1777.out‚0 cpu‚25.357122340671516 disk.rbd3.in‚0 disk.sda.out‚ 360.2734375 disk.rbd0.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.07146167755127 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 40749.921875$ memory_percent‚93.70760917663574# proc.memory.rssMB‚ 2796.98828125tHýt2 *- _wandbruntime‚9209  _runtime‚9209ZBY Ê¡*Ä _step‚971 ' lr-AdamW/pg1‚2.2297157033993674e-07 ' lr-AdamW/pg2‚2.2297157033993674e-07 trainer/global_step‚11849 % _timestamp‚1.7884571177545514e+09 _runtime‚9209.308926339T²ÿÅÏ¢*É % _timestamp‚1.7884571177545514e+09 _runtime‚9209.308926339 _step‚971 ' lr-AdamW/pg1‚2.2297157033993674e-07 ' lr-AdamW/pg2‚2.2297157033993674e-07 trainer/global_step‚11849Ë?d>2£*- _wandbruntime‚9209  _runtime‚9209Bຒ©'¤*æO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270321664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56561753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626405 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464149593600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚972 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564794880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9209.309909771 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172224200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752057413586432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157046479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56562209 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406712067072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646443 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9920564794880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100484270 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10406540054528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850603 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698603595956224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonit¿âþCor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268ž“òñ'¥*ëO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688683031161344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406712067072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646443 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9920564794880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100484270 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10406540054528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850603 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698603595956224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100484268 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270321664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56561753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223626405 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762464149593600 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚972 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564794880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670641209193472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9209.309909771 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172224200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752057413586432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157046479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56562209 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Ìœ¦Ïá2¦*- _wandbruntime‚9209  _runtime‚9209ÒÎÙ,ö&§*ðM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698661442801664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157058680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9922040598528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56566683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526655871488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9921384222720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407567607296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56566244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚11849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407410288128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442801664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688739402203136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0™} û&¨*õM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56566244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172238214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 trainer/global_step‚11849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407410288128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172238216 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442801664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100492436 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688739402203136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698661442801664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157058680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9922040598528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172238216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56566683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚762526655871488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9921384222720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395883073 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407567607296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844Íy BZ2©*- _wandbruntime‚9209  _runtime‚9209É»¾Zª*Ô 0loss/FBt4/assignment_loss‚0.2523975036191566 /loss/FRt4/detection_loss‚3.1515540927648544 1loss/SRqqt3/detection_loss‚1.5838068013841455 0loss/FRt3/assignment_loss‚3.0859531153712356 1loss/FBt3/assignment_loss‚0.12580957869067788 /loss/FRt3/detection_loss‚2.0122640760321366 /loss/FBt2/detection_loss‚0.5484583962386531 2loss/SRqqt4/assignment_loss‚1.6225492681922584 0loss/FBt2/assignment_loss‚0.2036283940506974 -loss/FRt2/detection_loss‚1.67516049383967 1loss/SRqqt1/detection_loss‚1.5001068030502283 0loss/FRt1/assignment_loss‚2.9044722670163865 .loss/FRt2/assignment_loss‚1.19573516536642 2loss/SRqqt3/assignment_loss‚1.5509927819398317 /loss/FRt1/detection_loss‚2.5310679085743732 .loss/FBt3/detection_loss‚2.890162408351898 _runtime‚9209.977574214 _step‚974 epoch‚9 trainer/global_step‚11849 /loss/FRt4/assignment_loss‚3.326658472418785 2loss/SRqqt1/assignment_loss‚1.5374308869827027 1loss/FBt1/assignment_loss‚0.15926880121514841 .loss/FBt1/detection_loss‚3.020444169640541 %loss/total_loss‚41.85950480633164 1loss/SRqqt2/detection_loss‚1.0474432318929643 1loss/SRqqt4/detection_loss‚2.5637019502705543 .loss/FBt4/detection_loss‚2.402349041266875 2loss/SRqqt2/assignment_loss‚0.9680891981614488 % _timestamp‚1.7884571184232397e+09%›ÑVå«*Ù 2loss/SRqqt2/assignment_loss‚0.9680891981614488 % _timestamp‚1.7884571184232397e+09 0loss/FBt4/assignment_loss‚0.2523975036191566 /loss/FRt4/detection_loss‚3.1515540927648544 1loss/SRqqt3/detection_loss‚1.5838068013841455 0loss/FRt3/assignment_loss‚3.0859531153712356 1loss/FBt3/assignment_loss‚0.12580957869067788 /loss/FRt3/detection_loss‚2.0122640760321366 /loss/FBt2/detection_loss‚0.5484583962386531 2loss/SRqqt4/assignment_loss‚1.6225492681922584 0loss/FBt2/assignment_loss‚0.2036283940506974 -loss/FRt2/detection_loss‚1.67516049383967 1loss/SRqqt1/detection_loss‚1.5001068030502283 0loss/FRt1/assignment_loss‚2.9044722670163865 .loss/FRt2/assignment_loss‚1.19573516536642 2loss/SRqqt3/assignment_loss‚1.5509927819398317 /loss/FRt1/detection_loss‚2.5310679085743732 .loss/FBt3/detection_loss‚2.890162408351898 _runtime‚9209.977574214 _step‚974 epoch‚9 trainer/global_step‚11849 /loss/FRt4/as8Óëzsignment_loss‚3.326658472418785 2loss/SRqqt1/assignment_loss‚1.5374308869827027 1loss/FBt1/assignment_loss‚0.15926880121514841 .loss/FBt1/detection_loss‚3.020444169640541 %loss/total_loss‚41.85950480633164 1loss/SRqqt2/detection_loss‚1.0474432318929643 1loss/SRqqt4/detection_loss‚2.5637019502705543 .loss/FBt4/detection_loss‚2.402349041266875ÎùÓ‚5’¬*‚(:‡ žáæÔñíßÕ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.689 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è<ÓÝL­*‚(:Á žáæÔȱ™Ø disk.dm-1.out‚ 360.65234375 disk.sda.in‚ 0.0234375 disk.sda.out‚ 360.65234375$ disk./.usageGB‚1109.533847808838 disk.rbd0.in‚0" disk.dm-0.out‚40751.2939453125 disk.rbd3.out‚0$ memory_percent‚93.70896816253662 proc.cpu.threads‚17! disk.rbd2.out‚316.45556640625' disk.nvme0c0n1.out‚40751.2939453125 disk.sda2.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1030.72265625 disk.sda3.in‚ 0.0234375 disk.dm-0.in‚ 1560.38671875 disk.rbd1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2796.98828125) proc.memory.percent‚17.07146167755127 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 cpu‚25.356598552440662 disk.rbd1.out‚ 4.599609375 network.recv‚ 3.352049e+06 disk.sda3.out‚ 360.65234375% disk.nvme0n1.out‚40751.2939453125 disk.sda1.out‚0 network.sent‚ 2.9216515e+07) disk./.usagePercent‚74.46174748735773# disk.nvme0c0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0234375ˆw°qœ®*‚(:‘ ­áæÔ­·­Õ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.960000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÝõÃJ¯*‚(:¿ ­áæÔœªáØ disk.rbd1.out‚ 4.599609375( proc.memory.availableMB‚ 1031.0859375 disk.dm-1.out‚ 361.24609375 disk.rbd3.out‚0 network.recv‚ 3.358797e+06% disk./.usageGB‚1109.5339241027832 disk.rbd2.out‚316.6201171875# disk.nvme0c0n1.in‚ 1560.38671875) disk./.usagePercent‚74.46175260750842 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.0234375 disk.rbd0.in‚0 disk.rbd3.in‚0$ memory_percent‚93.70675086975098 proc.cpu.threads‚17 disk.rbd0.out‚0' disk.nvme0c0n1.out‚40751.6767578125 disk.dm-0.in‚ 1560.38671875" disk.dm-0.out‚40751.6767578125 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2796.98828125 network.sent‚ 2.9324853e+07 disk.sda3.out‚ 361.24609375! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.out‚0) proc.memory.percent‚17.07146167755127 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0234375 disk.sda.out‚ 361.24609375 cpu‚25.35612523047257% disk.nvme0n1.out‚40751.6767578125 disk.sda.in‚ 0.0234375\”“°*‚(:ˆ ¼áæÔˆ£«Õ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LPµM±*‚(: ¼áæÔ×ÈÁØ disk.rbd3.in‚0 disk.sda.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875& disk.nvme0n1.out‚40760.11474609375 disk.dm-1.out‚ 361.65234375 disk.sda1.out‚0( proc.memory.availableMB‚ 1030.4453125# proc.memory.rssMB‚ 2796.98828125 network.sent‚ 2.9331097e+07) disk./.usagePercent‚74.46228305511936 disk.sda3.in‚ 0.0234375 disk.sda3.out‚ 361.65234375 disk.rbd2.out‚316.6201171875 disk.dm-0.in‚ 1560.38671875 disk.rbd1.in‚0% disk./.usageGB‚1109.5418281555176 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.0234375) proc.memory.percent‚17.07146167755127 cpu‚25.35563107069488 disk.sda2.in‚0 disk.sda.out‚ 361.65234375# disk.dm-0.out‚40760.11474609375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.363562e+06! disk.nvme0n1.in‚ 1560.38671875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.599609375$ memory_percent‚93.71066093444824( disk.nvme0c0n1.out‚40760.11474609375 disk.drbd1777.out‚0Áwö—2²*- _wandbruntime‚9244  _runtime‚9244§\~uɳ*à ' lr-AdamW/pg1‚1.4007070412152667e-07 ' lr-AdamW/pg2‚1.4007070412152667e-07 trainer/global_step‚11899 % _timestamp‚1.7884571533896458e+09 _runtime‚ 9244.94404604 _step‚975–DQδ*È ' lr-AdamW/pg2‚1.4007070412152667e-07 trainer/global_step‚11899 % _timestamp‚1.7884571533896458e+09 _runtime‚ 9244.94404604 _step‚975 ' lr-AdamW/pg1‚1.4007070412152667e-07Ï:¶œ2µ*- _wandbruntime‚9244  _runtime‚9244”·²jì'¶*æO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683311949347418 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚683311277870302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206963200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908069888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚9244.944966814 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚691500683103744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673386415721472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172925000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10449728635392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224552457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9961550600704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224552457 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157682556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9961550600704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691500701106688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572495 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9962206963200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755139950469632 trainer/global_step‚11899 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157683004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589703045120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100892668 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172924998 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884571533904278e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9925533625946 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0öØ3ñ'·*ëO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172924998 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884571533904278e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9925533625946 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683311949347418 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚683311277870302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9962206963200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701462908069888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 _runtime‚9244.944966814 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172924998 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673386398682112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚691500683103744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673386415721472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755139974409728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172925000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10449728635392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224552457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9961550600704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224552457 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157682556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9961550600704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691500701106688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224572495 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9962206963200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691500701106688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755139950469632 trainer/global_step‚11899 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157683004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589703045120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100892668ÐhI°á2¸*- _wandbruntime‚9245  _runtime‚9245šòtso¹*ïM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691557036142592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691557054145536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765652218330112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765652028682240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚977 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435826 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061118 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026404352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56794914 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9925695623154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590975 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788457154068702e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701520080549888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚9245.623715479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10450593182720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691557036142592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚11899 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172939016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batÕo5¹† ch_end/allocated_bytes.all.freed‚701519406171136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673441302812672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397509925 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157695296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397529991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570911 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157695752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ƒ`ˆú&º*ôM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚11899 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172939016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701519406171136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673441302812672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397509925 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157695296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397529991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570911 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157695752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172939016 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691557036142592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683367669913150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691557054145536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765652218330112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765652028682240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚977 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435826 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061118 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026404352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56794914 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9925695623154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590975 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788457154068702e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701520080549888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚9245.623715479 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10450593182720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691557036142592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925Ñ>)«2»*- _wandbruntime‚9245  _runtime‚9245È'ï_¼*Ù /loss/FRt3/assignment_loss‚2.845566439385317 2loss/SRqqt4/assignment_loss‚1.5122978763805854 /loss/FRt4/assignment_loss‚2.935135233402252 %loss/total_loss‚39.60958072395691 _runtime‚9245.625220834 2loss/SRqqt2/assignment_loss‚0.8398105349165924 /loss/FRt3/detection_loss‚2.2149586537662818 .loss/FRt4/detection_loss‚2.695685371330806 1loss/SRqqt1/detection_loss‚1.6203432708978653 1loss/SRqqt3/detection_loss‚1.6401122094920049 trainer/global_step‚11899 2loss/SRqqt3/assignment_loss‚1.6344149604869576 1loss/FBt3/assignment_loss‚0.08513575325855527 2loss/FBt4/assignment_loss‚0.017177292320411652 1loss/SRqqt2/detection_loss‚0.9343116820197214 1loss/SRqqt4/detection_loss‚2.0267997263251125 /loss/FBt3/detection_loss‚3.4585571214556694 .loss/FRt1/assignment_loss‚2.50804103910923 2loss/SRqqt1/assignment_loss‚0.9578730774857104 .loss/FRt1/detection_loss‚2.346620675176382 .loss/FBt1/detection_loss‚3.037011706829071 /loss/FBt4/detection_loss‚2.4629195978244147 % _timestamp‚1.7884571540709288e+09 /loss/FRt2/detection_loss‚1.6259430623054505 1loss/FBt1/assignment_loss‚0.28211387537667176 1loss/FBt2/assignment_loss‚0.09134897314006546 0loss/FBt2/detection_loss‚0.48803226427593227 epoch‚9 _step‚978 0loss/FRt2/assignment_loss‚1.3493703269958497Íä4d½*Þ 0loss/FRt2/assignment_loss‚1.3493703269958497 /loss/FRt3/assignment_loss‚2.845566439385317 2loss/SRqqt4/assignment_loss‚1.5122978763805854 /loss/FRt4/assignment_loss‚2.935135233402252 %loss/total_loss‚39.60958072395691 _runtime‚9245.625220834 2loss/SRqqt2/assignment_loss‚0.8398105349165924 /loss/FRt3/detection_loss‚2.2149586537662818 .loss/FRt4/detection_loss‚2.695685371330806 1loss/SRqqt1/detection_loss‚1.6203432708978653 1loss/SRqqt3/detection_loss‚1.6401122094920049 trainer/global_step‚11899 2loss/SRqqt3/assignment_loss‚1.6344149604869576 1loss/FBt3/assignment_loss‚0.08513575325855527 2loss/FBt4/assignment_loss‚0.017177292320411652 1loss/SRqqt2/detection_loss‚0.9343116820197214 1loss/SRqqt4/detection_loss‚2.0267997263251125 /loss/FBt3/detection_loss‚3.4585571214556694 .loss/FRt1/assignment_loss‚2.50804103910923 2loss/SRqqt1/assignment_loss‚0.9578730774857104 .loss/FRt1/detection_loss‚2.346620675176382 .loss/FBt1/detection_loss‚3.037011706829071 /loss/FBt4/detection_loss‚2.4629195978244147 % _timestamp‚1.7884571540709288e+09 /loss/FRt2/detection_loss‚1.6259430623054505 1loss/FBt1/assignment_loss‚0.28211387537667176 1loss/FBt2/assignment_loss‚0.09134897314006546 0loss/FBt2/detection_loss‚0.48803226427593227 epoch‚9 _step‚978ÒÕýHÎ’¾*‚(:‡ ËáæÔЃÑÕ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚90.143' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾?<‰M¿*‚(: ËáæÔá’›Ø network.recv‚ 3.367333e+06% disk./.usageGB‚1109.5419082641602 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda.in‚ 0.0234375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd3.in‚0# memory_percent‚93.7075138092041 network.sent‚ 2.9336374e+07 disk.sda.out‚ 362.10546875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚40760.47021484375& disk.nvme0n1.out‚40760.47021484375 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2796.98828125) proc.memory.percent‚17.07146167755127 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1560.38671875# disk.dm-0.out‚40760.47021484375 disk.dm-1.out‚ 362.10546875 cpu‚25.355138765434226) disk./.usagePercent‚74.46228843127759 disk.sda3.in‚ 0.0234375 disk.rbd0.in‚0 disk.sda3.out‚ 362.10546875 disk.rbd0.out‚0 disk.rbd2.out‚318.3349609375 disk.dm-1.in‚ 0.0234375 disk.rbd3.out‚0( proc.memory.availableMB‚ 1030.9609375Á°TÀ*‚(:’ ÚáæÔ‹ÙÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚91.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.251800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûì‰EÁ*‚(:º ÚáæÔÜú«Ù network.sent‚ 2.9445239e+07# disk.nvme0c0n1.in‚ 1560.38671875& disk.nvme0c0n1.out‚40760.994140625 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 disk.rbd1.out‚ 4.599609375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0! disk.dm-0.out‚40760.994140625 disk.dm-1.out‚ 362.6328125 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.0234375$ memory_percent‚93.70920658111572 network.recv‚ 3.374292e+06 disk.sda.out‚ 362.6328125# proc.memory.rssMB‚ 2796.98828125$ disk.nvme0n1.out‚40760.994140625 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.35466688250728 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0234375) proc.memory.percent‚17.07146167755127% disk./.usageGB‚1109.5409545898438! disk.nvme0n1.in‚ 1560.38671875 disk.sda3.in‚ 0.0234375 disk.rbd2.out‚318.3349609375) proc.memory.availableMB‚ 1030.68359375) disk./.usagePercent‚74.46222442939401 disk.sda2.out‚0 disk.sda3.out‚ 362.6328125 disk.rbd0.out‚0 disk.rbd2.in‚0±82Â*- _wandbruntime‚9280  _runtime‚9280j…/kÇÃ*Á & lr-AdamW/pg1‚7.634294084639921e-08 & lr-AdamW/pg2‚7.634294084639921e-08 trainer/global_step‚11949 % _timestamp‚1.7884571892499993e+09 _runtime‚ 9280.80444818 _step‚979TK]ÌÌÄ*Æ & lr-AdamW/pg1‚7.634294084639921e-08 & lr-AdamW/pg2‚7.634294084639921e-08 trainer/global_step‚11949 % _timestamp‚1.7884571892499993e+09 _runtime‚ 9280.80444818 _step‚979ÓJÂ"2Å*- _wandbruntime‚9280  _runtime‚9280ݦÍð'Æ*êO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871452672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002536406528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704321545818112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11949 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101301068 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704321545818112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192769024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768715127225344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225498547 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225498547 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9280.805764239 _step‚980 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225478509 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158318614 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124347 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768715316886528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301070 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704321545818112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10003192769024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768715127225344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225498547 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225498547 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173625800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9280.805764239 _step‚980 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225478509 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676131605210112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158318614 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚694318353049088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124347 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768715316886528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676131622249472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758222511292928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173625800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301070 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 $ _timestamp‚1.788457189919761e+09 _runtime‚9281.475079129 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768777828282368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493504738304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158330850 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚981 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚694374688084992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777642828800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225496963 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355834368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443Ä$5êþ&Ê*øM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚758284138090496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686153040303234 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225496963 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚694374706087936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004012210176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚694374706087936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10003355834368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758284162030592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173639816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173639816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚686153711780558 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676186509340672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 $ _timestamp‚1.788457189919761e+09 _runtime‚9281.475079129 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225517027 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768777828282368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493504738304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158330850 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚981 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚694374688084992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768777642828800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225496963 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10003355834368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÕlJÊ2Ë*- _wandbruntime‚9281  _runtime‚9281‚AUÂ]Ì*× 0loss/FRt3/assignment_loss‚2.7596983551979064 2loss/SRqqt2/assignment_loss‚0.8639098726341036 /loss/FBt2/detection_loss‚0.7620711310135909 /loss/FRt2/detection_loss‚1.7510373857803643 1loss/SRqqt2/detection_loss‚1.0610266136129698 1loss/SRqqt3/detection_loss‚2.0726121889220344 _step‚982 2loss/SRqqt1/assignment_loss‚1.1955718023569457 /loss/FRt1/detection_loss‚2.1302927522496744 trainer/global_step‚11949 1loss/FBt4/assignment_loss‚0.32585264892051263 1loss/FBt3/assignment_loss‚0.16786879176894823 .loss/FBt3/detection_loss‚2.812444269657135 .loss/FRt3/detection_loss‚2.110049766302109 %loss/total_loss‚40.10129897410289 _runtime‚9281.476537336 0loss/FRt4/assignment_loss‚2.4135368919372557 0loss/SRqqt3/assignment_loss‚1.05377353462908 2loss/SRqqt4/assignment_loss‚2.0658803316764534 1loss/FBt2/assignment_loss‚0.18376072349330974 /loss/FBt1/detection_loss‚2.4689384140074253 /loss/FRt4/detection_loss‚3.0778255438804627 .loss/FBt4/detection_loss‚2.449054939406259 epoch‚9 % _timestamp‚1.7884571899222016e+09 0loss/FRt1/assignment_loss‚2.7434092367237266 0loss/FRt2/assignment_loss‚1.3262984541555245 1loss/FBt1/assignment_loss‚0.30809042102094963 /loss/SRqqt1/detection_loss‚1.55152722501329 1loss/SRqqt4/detection_loss‚2.4467676797428646$"2bÍ*Ü 0loss/FRt3/assignment_loss‚2.7596983551979064 2loss/SRqqt2/assignment_loss‚0.8639098726341036 /loss/FBt2/detection_loss‚0.7620711310135909 /loss/FRt2/detection_loss‚1.7510373857803643 1loss/SRqqt2/detection_loss‚1.0610266136129698 1loss/SRqqt3/detection_loss‚2.0726121889220344 _step‚982 2loss/SRqqt1/assignment_loss‚1.1955718023569457 /loss/FRt1/detection_loss‚2.1302927522496744 trainer/global_step‚11949 1loss/FBt4/assignment_loss‚0.32585264892051263 1loss/FBt3/assignment_loss‚0.16786879176894823 .loss/FBt3/detection_loss‚2.812444269657135 .loss/FRt3/detection_loss‚2.110049766302109 %loss/total_loss‚40.10129897410289 _runtime‚9281.476537336 0loss/FRt4/assignment_loss‚2.4135368919372557 0loss/SRqqt3/assignment_loss‚1.05377353462908 2loss/SRqqt4/assignment_loss‚2.0658803316764534 1loss/FBt2/assignment_loss‚0.18376072349330974 /loss/FBt1/detection_loss‚2.4689384140074253 /loss/FRt4/detection_loss‚3.0778255438804627 .loss/FBt4/detection_loss‚2.449054939406259 epoch‚9 % _timestamp‚1.7884571899222016e+09 0loss/FRt1/assignment_loss‚2.7434092367237266 0loss/FRt2/assignment_loss‚1.3262984541555245 1loss/FBt1/assignment_loss‚0.30809042102094963 /loss/SRqqt1/detection_loss‚1.55152722501329 1loss/SRqqt4/detection_loss‚2.4467676797428646ÖÊŒw·“Î*‚(:ˆ éáæÔ²ÑâÕ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á0Ò“LÏ*‚(:Á éáæÔÆ’Ø) disk./.usagePercent‚74.47068394235546 disk.dm-1.in‚ 0.0234375( proc.memory.availableMB‚ 1030.5703125 disk.rbd1.in‚0 disk.rbd3.in‚0 network.recv‚ 3.37839e+06$ disk./.usageGB‚1109.667007446289 disk.sda3.in‚ 0.0234375 disk.rbd0.in‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd2.out‚318.4208984375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.sda.in‚ 0.0234375 proc.cpu.threads‚17 disk.rbd0.out‚0& disk.nvme0n1.out‚40858.57763671875# disk.dm-0.out‚40858.57763671875 disk.rbd1.out‚ 4.599609375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 363.10546875$ memory_percent‚93.70989799499512 disk.sda1.in‚0( disk.nvme0c0n1.out‚40858.57763671875 disk.rbd3.out‚0 disk.sda.out‚ 363.10546875) proc.memory.percent‚17.07146167755127 cpu‚25.354155980717458 network.sent‚ 2.9450881e+07 disk.sda3.out‚ 363.10546875# proc.memory.rssMB‚ 2796.98828125¡¨ÚÞ“Ð*‚(:ˆ øáæÔ–ĪÕ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚106.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œe‡Ç¬Ñ*‚(:½ øáæÔäÐü×% disk./.usageGB‚1109.7924423217773 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚318.4208984375 disk.drbd1777.in‚0 cpu‚25.353640934837568 proc.cpu.threads‚17 disk.sda.in‚ 0.0234375( disk.nvme0c0n1.out‚40859.73193359375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1560.38671875# disk.dm-0.out‚40859.73193359375 disk.dm-1.out‚ 363.8515625 disk.rbd1.in‚0 network.sent‚ 2.9560004e+07 disk.sda3.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875& disk.nvme0n1.out‚40859.73193359375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 4.599609375% proc.memory.availש‘dœableMB‚ 1030.4375) proc.memory.percent‚17.07146167755127 network.recv‚ 3.385643e+06 disk.sda3.out‚ 363.8515625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.out‚0$ memory_percent‚93.71070861816406# proc.memory.rssMB‚ 2796.98828125 disk.dm-1.in‚ 0.0234375) disk./.usagePercent‚74.47910198209635 disk.sda.out‚ 363.8515625 disk.sda2.in‚0^ŽíÎ’Ò*‚(:‡ ‡âæÔÔžïÕ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚34.0 gpu.0.powerWatts‚91.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 úЛGÓ*‚(:¼ ‡âæÔèÖý× network.recv‚ 3.389171e+06 disk.dm-1.in‚ 0.0234375 disk.sda1.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1560.38671875 disk.rbd1.in‚0# disk.dm-0.out‚40957.25146484375 disk.sda2.in‚0) proc.memory.percent‚17.07146167755127! disk.nvme0n1.in‚ 1560.38671875 cpu‚25.353153227984265 disk.dm-1.out‚ 364.28125 disk.rbd1.out‚ 4.599609375 disk.rbd3.in‚0) disk./.usagePercent‚74.47907126119225% disk./.usageGB‚1109.7919845581055 disk.sda.in‚ 0.0234375 disk.sda3.in‚ 0.0234375 disk.sda3.out‚ 364.28125 disk.dm-0.in‚ 1560.38671875( disk.nvme0c0n1.out‚40957.25146484375! disk.rbd2.out‚318.60498046875& disk.nvme0n1.out‚40957.25146484375 disk.rbd3.out‚0 network.sent‚ 2.9565228e+07 disk.sda2.out‚0 disk.sda.out‚ 364.28125 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ memory_percent‚93.70977878570557) proc.memory.availableMB‚ 1030.58984375# proc.memory.rssMB‚ 2796.98828125’6Ø%2Ô*- _wandbruntime‚9316  _runtime‚9316S¦çfÈÕ* _runtime‚9316.329623036 _step‚983 & lr-AdamW/pg1‚3.180156128487796e-08 & lr-AdamW/pg2‚3.180156128487796e-08 trainer/global_step‚11999 % _timestamp‚1.7884572247752204e+09'ÖÍÖ*Ç & lr-AdamW/pg1‚3.180156128487796e-08 & lr-AdamW/pg2‚3.180156128487796e-08 trainer/global_step‚11999 % _timestamp‚1.7884572247752204e+09 _runtime‚9316.329623036 _step‚983×æAAo2×*- _wandbruntime‚9316  _runtime‚9316¦€_çò'Ø*ìO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082234 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535531662848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178574848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158954384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚9316.330509285 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174326600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 trainer/global_step‚11999 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158954823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549867006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2Š•?Z÷'Ù*ñO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522212352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174326600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226424599 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226404561 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707180183566336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57244916 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚9316.330509285 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174326600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404561 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 trainer/global_step‚11999 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158954823 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549867006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688884033082234 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535531662848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10044178574848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158954384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158967074 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688939082170642 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10536546710528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚985 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707236681667584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158967514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚697192340027392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226423015 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771903049684992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10008020739934 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101717638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761366698913792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903245624320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226443079 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226443079 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341640192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10536374711296 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚11999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998016000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884572254475877e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707237356046336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771903245624320 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226443079 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226443079 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044341640192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10536374711296 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚11999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044998016000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884572254475877e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707237356046336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158967074 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688939082170642 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10536546710528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚985 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚707236681667584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158967514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚697192340027392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ÙO2Ý*- _wandbruntime‚9317  _runtime‚9317;~®ZÞ*Ô 0loss/FRt3/assignment_loss‚2.9362147053082785 .loss/FRt1/detection_loss‚2.403349275174348 .loss/FRt2/detection_loss‚1.845536521497976 .loss/FRt4/detection_loss‚2.802889856852983 1loss/FBt3/assignment_loss‚0.24385607428477366 1loss/SRqqt2/detection_loss‚0.8552556933443013 /loss/FBt2/detection_loss‚0.6732046787568013 .loss/SRqqt1/detection_loss‚1.7856418689092 0loss/FRt2/assignment_loss‚1.2438340948687658 2loss/SRqqt3/assignment_loss‚1.2776214012503624 1loss/FBt1/assignment_loss‚0.25925169754227684 1loss/FBt4/assignment_loss‚0.22153826932688908 /loss/FBt1/detection_loss‚2.5703393697738646 /loss/FBt4/detection_loss‚2.6597710805279866 0loss/FRt4/assignment_loss‚3.3883414895910966 1loss/SRqqt1/assignment_loss‚1.347024335283398 epoch‚9 % _timestamp‚1.7884572254498432e+09 0loss/FRt1/assignment_loss‚2.6538029351960057 1loss/SRqqt4/assignment_loss‚2.099682819417545 .loss/FRt3/detection_loss‚2.037040153877777 trainer/global_step‚11999 _runtime‚ 9317.00415749 _step‚986 2loss/SRqqt2/assignment_loss‚0.9035561520018076 1loss/FBt2/assignment_loss‚0.16349653727428823 1loss/SRqqt3/detection_loss‚1.6852279328554869 1loss/SRqqt4/detection_loss‚2.3847493393080574 /loss/FBt3/detection_loss‚2.8897439738114676 %loss/total_loss‚41.33097025603574RêÂ_ß*Ù .loss/SRqqt1/detection_loss‚1.7856418689092 0loss/FRt2/assignment_loss‚1.2438340948687658 2loss/SRqqt3/assignment_loss‚1.2776214012503624 1loss/FBt1/assignment_loss‚0.25925169754227684 1loss/FBt4/assignment_loss‚0.22153826932688908 /loss/FBt1/detection_loss‚2.5703393697738646 /loss/FBt4/detection_loss‚2.6597710805279866 0loss/FRt4/assignment_loss‚3.3883414895910966 1loss/SRqqt1/assignment_loss‚1.347024335283398 epoch‚9 % _timestamp‚1.7884572254498432e+09 0loss/FRt1/assignment_loss‚2.6538029351960057 1loss/SRqqt4/assignment_loss‚2.099682819417545 .loss/FRt3/detection_loss‚2.037040153877777 trainer/global_step‚11999 _runtime‚ 9317.00415749 _step‚986 2loss/SRqqt2/assignment_loss‚0.9035561520018076 1loss/FBt2/assignment_loss‚0.16349653727428823 1loss/SRqqt3/detection_loss‚1.6852279328554869 1loss/SRqqt4/detection_loss‚2.3847493393080574 /loss/FBt3/detection_loss‚2.8897439738114676 %loss/total_loss‚41.33097025603574 0loss/FRt3/assignment_loss‚2.9362147053082785 .loss/FRt1/detection_loss‚2.403349275174348 .loss/FRt2/detection_loss‚1.845536521497976 .loss/FRt4/detection_loss‚2.802889856852983 1loss/FBt3/assignment_loss‚0.24385607428477366 1loss/SRqqt2/detection_loss‚0.8552556933443013 /loss/FBt2/detection_loss‚0.6732046787568013Úšºžà*‚(:“ –âæÔ‰¹¹Õ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.999' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.599800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷æÆLá*‚(:Á –âæÔ¼¡ÐØ$ memory_percent‚93.70975494384766 cpu‚25.352636877484212 disk.sda2.in‚0 disk.dm-1.out‚ 364.60546875 disk.sda3.out‚ 364.60546875& proc.memory.availableMB‚ 1030.59375 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1560.38671875) proc.memory.percent‚17.07146167755127 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 3.393585e+06 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 364.60546875 disk.sda1.out‚0 disk.sda3.in‚ 0.0234375( disk.nvme0c0n1.out‚41054.85791015625 network.sent‚ 2.9571405e+07% disk./.usageGB‚1110.0419273376465 disk.drbd1777.out‚0( disk./.usagePercent‚74.4958451308432! disk.nvme0n1.in‚ 1560.38671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2796.98828125 disk.rbd2.in‚0! disk.rbd2.out‚318.60498046875 disk.dm-0.in‚ 1560.38671875# disk.dm-0.out‚41054.85791015625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.0234375& disk.nvme0n1.out‚41054.85791015625 disk.dm-1.in‚ 0.0234375î…ôU“â*‚(:ˆ ¥âæÔ›…ºÕ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.299' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´R`jNã*‚(:à ¥âæÔÀçóØ( disk.nvme0c0n1.out‚41055.21728515625$ memory_percent‚93.71061325073242) proc.memory.percent‚17.07146167755127 cpu‚25.352177560516015) disk./.usagePercent‚74.49584897095622 disk.sda1.out‚0! disk.rbd2.out‚320.31982421875# disk.nvme0c0n1.in‚ 1560.38671875! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.out‚0 disk.dm-1.out‚ 365.27734375 disk.sda3.in‚ 0.0234375# proc.memory.rssMB‚ 2796.98828125 disk.sda1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚41055.21728515625# disk.dm-0.out‚41055.21728515625 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.9679733e+07% disk./.usageGB‚1110.0419845581055 disk.rbd3.out‚0 disk.sda.in‚ 0.0234375 network.recv‚ 3.400221e+06 disk.sda3.out‚ 365.27734375 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.0234375 disk.rbd1.out‚ 4.599609375' proc.memory.availableMB‚ 1030.453125 disk.rbd0.in‚0 disk.sda.out‚ 365.27734375 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.in‚0 disk.rbd3.in‚0«º|h2ä*- _wandbruntime‚9352  _runtime‚9352á)pÃÇå*Á & lr-AdamW/pg1‚6.455847793719503e-09 & lr-AdamW/pg2‚6.455847793719503e-09 trainer/global_step‚12049 % _timestamp‚1.7884572606672018e+09 _runtime‚ 9352.22156265 _step‚987ƒuëIÌæ*Æ & lr-AdamW/pg1‚6.455847793719503e-09 & lr-AdamW/pg2‚6.455847793719503e-09 trainer/global_step‚12049 % _timestamp‚1.7884572606672018e+09 _runtime‚ 9352.22156265 _step‚987ÛßÍå"2ç*- _wandbruntime‚9352  _runtime‚9352{ †Îò'è*ìO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57472751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578683419136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710038821314560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚699953638930944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227330613 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9352.222445794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159591080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10578855431680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 % _timestamp‚1.7884572606679006e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175027398 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227330613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159590619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57473210 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175027400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚774966244538368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚691670074949642 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350651 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10085164380672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚988 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227350651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508018176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844SW‹&%é*ñO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 % _timestamp‚1.7884572606679006e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175027398 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227330613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159590619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57473210 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175027400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚774966244538368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681622035305472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚691670074949642 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175027398 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227350651 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10085164380672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚988 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227350651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚764387561119232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508018176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681622018266112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UG zé&ÒDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102117868 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57472751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578683419136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710038821314560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚699953638930944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227330613 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9352.222445794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159591080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10578855431680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102117870Ü]9©\2ê*- _wandbruntime‚9352  _runtime‚9352¦foÖû&ë*õM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚775028940589568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227349067 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700010009972736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175041414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028750941696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10048201641378 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681676922396672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402390481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515374 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579704792576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788457261333421e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579539084800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038050 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57477787 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159603825 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57477340 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚9352.888150737 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159603376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 trainer/global_step‚12049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016ÏEåô'ì*úM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681676922396672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402390481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚691725795515374 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579704792576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700009991969792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788457261333421e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579539084800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038050 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57477787 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681676939436032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159603825 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57477340 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚9352.888150737 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700009991969792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159603376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 trainer/global_step‚12049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚775028940589568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227349067 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700010009972736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175041414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775028750941696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10048201641378 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908ݦøÃF2í*- _wandbruntime‚9352  _runtime‚93521˶Ù_î*Ù 2loss/SRqqt1/assignment_loss‚1.4002789422520436 2loss/SRqqt4/assignment_loss‚2.0645464261372886 1loss/SRqqt1/detection_loss‚1.6287029897794127 _runtime‚9352.889160079 /loss/FRt1/detection_loss‚2.2057221566016474 /loss/FRt1/assignment_loss‚2.887366771697998 2loss/SRqqt3/assignment_loss‚1.3034757547851266 .loss/FRt3/detection_loss‚2.023424332737923 1loss/SRqqt3/detection_loss‚1.4584442825409873 1loss/FBt2/assignment_loss‚0.12037149515735751 epoch‚9 2loss/SRqqt2/assignment_loss‚0.9352661676193569 0loss/FBt4/assignment_loss‚0.4454843520758335 /loss/FBt1/detection_loss‚2.3727744966745377 1loss/FBt3/assignment_loss‚0.10148646675848536 /loss/FRt4/detection_loss‚2.8822026766579727 % _timestamp‚1.7884572613348777e+09 trainer/global_step‚12049 /loss/FRt2/assignment_loss‚1.348425005674362 /loss/FRt4/assignment_loss‚3.972431043098713 &loss/total_loss‚41.698870815924046 _step‚990 /loss/FRt2/detection_loss‚1.7850066667795181 /loss/FBt3/detection_loss‚3.1540508014815196 /loss/FBt4/detection_loss‚2.2358346202156762 0loss/FRt3/assignment_loss‚2.7505983412265778 1loss/FBt1/assignment_loss‚0.16560720923850644 0loss/SRqqt2/detection_loss‚1.330075721092084 1loss/SRqqt4/detection_loss‚2.4417978282328003 /loss/FBt2/detection_loss‚0.6854962674083143Æydï*Þ /loss/FRt2/assignment_loss‚1.348425005674362 /loss/FRt4/assignment_loss‚3.972431043098713 &loss/total_loss‚41.698870815924046 _step‚990 /loss/FRt2/detection_loss‚1.7850066667795181 /loss/FBt3/detection_loss‚3.1540508014815196 /loss/FBt4/detection_loss‚2.2358346202156762 0loss/FRt3/assignment_loss‚2.7505983412265778 1loss/FBt1/assignment_loss‚0.16560720923850644 0loss/SRqqt2/detection_loss‚1.330075721092084 1loss/SRqqt4/detection_loss‚2.4417978282328003 /loss/FBt2/detection_loss‚0.6854962674083143 2loss/SRqqt1/assignment_loss‚1.4002789422520436 2loss/SRqqt4/assignment_loss‚2.0645464261372886 1loss/SRqqt1/detection_loss‚1.6287029897794127 _runtime‚9352.889160079 /loss/FRt1/detection_loss‚2.2057221566016474 /loss/FRt1/assignment_loss‚2.887366771697998 2loss/SRqqt3/assignment_loss‚1.3034757547851266 .loss/FRt3/detection_loss‚2.023424332737923 1loss/SRqqt3/detection_loss‚1.4584442825409873 1loss/FBt2/assignment_loss‚0.12037149515735751 epoch‚9 2loss/SRqqt2/assignment_loss‚0.9352661676193569 0loss/FBt4/assignment_loss‚0.4454843520758335 /loss/FBt1/detection_loss‚2.3727744966745377 1loss/FBt3/assignment_loss‚0.10148646675848536 /loss/FRt4/detection_loss‚2.8822026766579727 % _timestamp‚1.7884572613348777e+09 trainer/global_step‚12049ÞClºÐžð*‚(:“ ´âæÔ¯š²Ö gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.847' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.169400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê0ůEñ*‚(:º ´âæÔ‘·ý×( disk./.usagePercent‚74.4958451308432% disk./.usageGB‚1110.0419273376465 disk.sda3.in‚ 0.0234375! disk.rbd2.out‚320.31982421875 disk.drbd1777.in‚0 network.recv‚ 3.405012e+06 disk.rbd1.out‚ 4.685546875 disk.sda.out‚ 365.828125 disk.sda1.in‚0) proc.memory.availableMB‚ 1030.21484375) proc.memory.percent‚17.07146167755127 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375 disk.dm-0.in‚ 1560.38671875$ memory_percent‚93.71206760406494 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 network.sent‚ 2.9686094e+07 disk.rbd2.in‚0# proc.memory.rssMB‚ 2796.98828125 cpu‚25.35167310659726 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0234375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚41153.3173828125" disk.dm-0.out‚41153.3173828125 disk.dm-1.out‚ 365.828125 disk.sda1.out‚0 disk.sda3.out‚ 365.828125# disk.nvme0c0n1.in‚ 1560.38671875% disk.nvme0n1.out‚41153.3173828125 disk.sda2.in‚0ÍT7/“ò*‚(:ˆ ÃâæÔÒ–ÊÕ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ÆÝ{Hó*‚(:½ ÃâæÔŸ¯¸Ø disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0$ memory_percent‚93.70989799499512# proc.memory.rssMB‚ 2796.98828125 cpu‚25.35118916444215 network.sent‚ 2.9794361e+07% disk./.usageGB‚1110.0425338745117 disk.dm-0.in‚ 1560.38671875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚41250.8486328125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1560.38671875 disk.drbd1777.out‚0 disk.sda3.out‚ 366.55859375) disk./.usagePercent‚74.49588583604114# disk.nvme0c0n1.in‚ 1560.38671875' disk.nvme0c0n1.out‚41250.8486328125 disk.dm-1.out‚ 366.55859375 disk.sda.out‚ 366.55859375 disk.sda1.out‚0( proc.memory.availableMB‚ 1030.5703125 disk.sda.in‚ 0.0234375 network.recv‚ 3.411258e+06" disk.dm-0.out‚41250.8486328125 disk.dm-1.in‚ 0.0234375 disk.rbd1.out‚ 4.685546875 disk.rbd3.in‚0 disk.sda3.in‚ 0.0234375 disk.rbd2.out‚ 320.48828125) proc.memory.percent‚17.07146167755127œEÅá“ô*‚(:ˆ ÒâæÔËŒÖÕ gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/À‘y8õ*‚(:­ ÒâæÔ—ûì×) disk./.usagePercent‚74.52957335545676 disk.sda2.in‚0 disk.sda.in‚ 0.0234375 disk.sda1.out‚0 disk.rbd2.out‚ 320.48828125 disk.rbd1.out‚ 4.748046875 disk.rbd3.out‚0 disk.rbd3.in‚0 network.sent‚ 2.980055e+07 disk.sda3.in‚ 0.0234375 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875 disk.sda2.out‚0$ memory_percent‚94.18904781341553 disk.rbd0.out‚0 disk.dm-0.out‚ 41251.140625( proc.memory.percent‚17.5492525100708 disk.dm-1.out‚ 367.0078125# disk.nvme0c0n1.in‚ 1560.38671875! disk.nvme0n1.in‚ 1560.38671875! disk.nvme0n1.out‚ 41251.140625 network.recv‚ 3.415775e+06% disk./.usageGB‚1110.5445022583008 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.0234375 disk.sda3.out‚ 367.0078125 disk.rbd1.in‚0 disk.sda.out‚ 367.0078125 disk.sda1.in‚0( proc.memory.availableMB‚ 952.06640625# proc.memory.rssMB‚ 2875.26953125 cpu‚25.355146764361884# disk.nvme0c0n1.out‚ 41251.140625_cÜPœö*‚(:‘ áâæÔ“¶íÖ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.083' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.816599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ÚS(~÷*‚(:¹ áâæÔÓïþ× disk.rbd2.out‚ 320.56640625 disk.dm-0.in‚ 1560.38671875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.in‚ 0.0234375 disk.rbd1.in‚0# memory_percent‚94.1870927810669 network.sent‚ 2.9806246e+07 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.0234375 disk.rbd2.in‚0# disk.nvme0c0n1.in‚ 1560.38671875! disk.nvme0n1.in‚ 1560.38671875 disk.sda2.in‚0# proc.memory.rssMB‚ 2875.26953125 cpu‚25.363011052991677% disk./.usageGB‚1110.5426635742188 disk.sda3.out‚ 367.57421875! disk.dm-0.out‚41348.708984375( proc.me;P¼ÎÆmory.percent‚17.5492525100708 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.0234375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚41348.708984375 disk.sda2.out‚0 disk.rbd1.out‚ 4.748046875( proc.memory.availableMB‚ 952.38671875) disk./.usagePercent‚74.52944995982524 disk.sda.out‚ 367.57421875$ disk.nvme0n1.out‚41348.708984375 disk.drbd1777.out‚0 disk.dm-1.out‚ 367.57421875 network.recv‚ 3.420085e+06ºfÝýø*‚(:’ ðâæÔà׶× gpu.0.gpu‚57.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.507400512695312* gpu.0.memoryAllocatedBytes‚ 10743775232 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.874399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßü06ù*‚(:« ðâæÔÁ„Ù disk.sda2.in‚0 disk.rbd1.out‚ 4.7802734375 network.recv‚ 3.423895e+06 disk.rbd2.in‚0 disk.sda.out‚368.1875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.9811666e+07 disk.sda3.out‚368.1875 disk.rbd0.out‚0! disk.dm-0.out‚41446.701171875# memory_percent‚94.1866397857666) disk./.usagePercent‚74.50687495944841% disk./.usageGB‚1110.2062797546387& disk.nvme0c0n1.out‚41446.701171875( proc.memory.percent‚17.5492525100708 disk.sda3.in‚ 0.0234375# disk.nvme0c0n1.in‚ 1560.38671875 disk.dm-1.out‚368.1875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚41446.701171875 disk.dm-1.in‚ 0.0234375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.0234375 disk.rbd2.out‚ 322.140625! disk.nvme0n1.in‚ 1560.38671875' proc.memory.availableMB‚ 952.4609375# proc.memory.rssMB‚ 2875.26953125 cpu‚25.370633552905424 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-0.in‚ 1560.38671875ŠŸ†'2ú*- _wandbruntime‚9429  _runtime‚9429‹`CY*û*ÓT OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 404597957 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 176131548 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚12089 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10136675672064 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581978 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚714204967776768 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚704068802937856 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚779403376027136 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚695701853648938 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚714205478609920 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 160653921 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚12089 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10136675672064 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581978 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚714204967776768 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚704068802937856 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚779403376027136 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560ßþ¹M:2ý*- _wandbruntime‚9430  _runtime‚9430äÈD#þ*ÝR bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718385002 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040798720 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162932224 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚695737511017834 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162932224 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10136547968512 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 176142354 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚768809028038656 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632449564672 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚704105632966656 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160664808 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚992 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚779441616887296 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10099227470610 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 176142354 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 trainer/global_step‚12089 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673765376 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457732 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57887004 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040798720 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 102777958 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928-QŠ-h)ÿ*âR M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 404616065 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 228457732 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚714242673765376 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 _runtime‚9430.018378248 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚779441477603328 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚704105614963712 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 176142356 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404600086 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 228473709 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10136547968512 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 160664962 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚685637792632832 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404616065 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚57887004 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10137040798720 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 102777958 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10099718385002 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10137040798720 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162932224 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚695737511017834 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚714242162932224 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 102777956 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚768809051978752 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10136547968512 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 176142354 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚704105614963712 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚768809028038656 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632449564672 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚704105632966656 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 160664808 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚992 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚779441616887296 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10099227470610 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 176142354 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 trainer/global_step‚12089 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚714242673765376 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 228457732 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15188133716583252 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2645901143550873 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5673941969871521 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11598599702119827 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6438480615615845 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.07692307978868484 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5297530889511108 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444443583488464 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_2_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12271638214588165 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 'particle/accuracy‚0.856342077255249 jet/accuracy_2_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1602383255958557 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10502862185239792 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14224138855934143 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5695345997810364 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697672963142395 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44887417554855347 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5826064348220825 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45443427562713623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4185393452644348 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6975609660148621 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013226337730884552 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 )loss/val_total_loss‚42.14930276358852 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005629203747957945 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821798205375671 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24053582549095154 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21647781133651733 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5238997936248779 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5264551043510437 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.130515918135643 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19911445677280426 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9794361591339111 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5019713640213013 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4457516372203827 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2045641988515854 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5396898984909058 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9834644794464111 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4547930657863617 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9756624102592468 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5611446499824524 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9767601490020752 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04611898586153984 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880925416946411 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1587301641702652 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.022260624915361404 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5503899455070496 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16322927176952362 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.464782178401947 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46189090609550476 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16843435168266296 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5622045993804932 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15738201141357422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5456712245941162 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5256702303886414 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03980669751763344 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_4‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14843828976154327 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5797064304351807 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14811520278453827 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655594 jet/accuracy_3_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49817588925361633 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4197562038898468 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29091373085975647 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908857345581055 particle/accuracy_2_of_2‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5263015627861023 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5924574732780457 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03951149433851242 particle/accuracy_2_of_3‚1 ,particle/sensitivity‚0.33898234367370605 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884761571884155 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9832519292831421 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +particle/specificity‚0.9520697593688965 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3348765969276428 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5990975499153137 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12899291515350342 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3324074149131775 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1949761062860489 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.568965494632721 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567901492118835 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22580084204673767 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35922619700431824 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6339285969734192 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333¦)%pI358168602 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4247775971889496 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 (particle/f_score‚0.42407962679862976 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2642804682254791 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5922566652297974 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5512802600860596 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866445064544678 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.43589746952056885 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5399146676063538 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18501602113246918 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9530081748962402 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18843233585357666 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6163052320480347 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 $ _timestamp‚1.788457351436096e+09 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19276337325572968 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5509949326515198 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03479752317070961 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14613953232765198 particle/accuracy_3_of_5‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2560602128505707 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9669539928436279 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5671296715736389 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46702510118484497 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006527612917125225 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.509147584438324 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6050140857696533 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017019400373101234 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9906768202781677 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5027843713760376 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10045550018548965 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27868157625198364 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3863636255264282 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4485805332660675 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11424409598112106 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5955325365066528 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18625105917453766 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32113736867904663 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 *jet/accuracy_1_of_3‚0.8857831358909607 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9275143146514893 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27831825613975525 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5509949326515198 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7777267694473267 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.023847319185733795 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209311962127686 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9854731559753418 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35750535130500793 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5732630491256714 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20334763824939728 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5638822317123413 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3729030191898346 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6148809790611267 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2642902433872223 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1629798710346222 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3814893066883087 _step‚993 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.598397433757782 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.06390224397182465 particle/accuracy_7_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36326098442077637 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.034770891070365906 *jet/accuracy_1_of_1‚0.5104520320892334 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5762362480163574 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18501602113246918 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5162213444709778 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.38055554032325745 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19179946184158325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5679560303688049 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1468568593263626 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20873157680034637 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9896695017814636 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14375901222229004 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3187499940395355 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.043600402772426605 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44384390115737915 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.12974271178245544 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3944006860256195 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24252872169017792 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40684524178504944 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_5_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.041220128536224365 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.016282642260193825 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15034161508083344 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6163150072097778 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.591005265712738 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5151230096817017 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358208656311 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 particle/accuracy_3_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023437581956386566 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 epoch‚9 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.587684154510498 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37273505330085754 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977625012397766 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5592939257621765 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.28998863697052 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1261243224143982 particle/accuracy_1_of_1‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10383116453886032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16509157419204712 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2929350733757019 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5812316536903381 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07189279794692993 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4636015295982361 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5362905859947205 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0)¹–/‚6„+„§ 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.9275143146514893 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27831825613975525 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5509949326515198 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7777267694473267 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.023847319185733795 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209311962127686 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9854731559753418 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35750535130500793 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5732630491256714 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20334763824939728 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5638822317123413 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3729030191898346 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6148809790611267 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2642902433872223 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1629798710346222 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3814893066883087 _step‚993 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.598397433757782 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.06390224397182465 particle/accuracy_7_of_7‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36326098442077637 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.034770891070365906 *jet/accuracy_1_of_1‚0.5104520320892334 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5762362480163574 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18501602113246918 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5162213444709778 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.38055554032325745 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19179946184158325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5679560303688049 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1468568593263626 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20873157680034637 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9896695017814636 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14375901222229004 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3187499940395355 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.043600402772426605 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44384390115737915 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.12974271178245544 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3944006860256195 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24252872169017792 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40684524178504944 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_5_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.041220128536224365 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.016282642260193825 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15034161508083344 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6163150072097778 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.591005265712738 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5151230096817017 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358208656311 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 particle/accuracy_3_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.023437581956386566 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 epoch‚9 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.587684154510498 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37273505330085754 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977625012397766 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5592939257621765 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.28998863697052 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1261243224143982 particle/accuracy_1_of_1‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10383116453886032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16509157419204712 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2929350733757019 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5812316536903381 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997860848903656 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07189279794692993 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4636015295982361 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5362905859947205 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 jet/accuracy_5_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5463659167289734 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 _runtime‚9442.992639361 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01901130937039852 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4243420958518982 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.09615384787321091 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5355294942855835 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5516474843025208 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4530303180217743 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30292433500289917 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21990743279457092 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19556331634521484 )jet/accuracy_2_of_3‚0.256112277507782 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22193469107151031 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11977852135896683 particle/accuracy_3_of_3‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02361193485558033 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11549708247184753 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9890708327293396 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066194295883179 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_7‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5395983457565308 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4448753893375397 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49885058403015137 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728071451187134 & PuË y™ ]rity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.986640453338623 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08561602234840393 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.221230149269104 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6489129066467285 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16116636991500854 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15188133716583252 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2645901143550873 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5673941969871521 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.11598599702119827 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6438480615615845 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.07692307978868484 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5297530889511108 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444443583488464 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_2_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12271638214588165 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 'particle/accuracy‚0.856342077255249 jet/accuracy_2_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1602383255958557 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10502862185239792 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14224138855934143 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5695345997810364 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697672963142395 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44887417554855347 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5826064348220825 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45443427562713623 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4185393452644348 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6975609660148621 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013226337730884552 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 )loss/val_total_loss‚42.14930276358852 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005629203747957945 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821798205375671 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24053582549095154 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21647781133651733 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5238997936248779 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5264551043510437 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.130515918135643 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19911445677280426 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9794361591339111 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5019713640213013 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4457516372203827 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2045641988515854 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5396898984909058 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3499999940395355 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9834644794464111 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4547930657863617 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9756624102592468 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5611446499824524 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9767601490020752 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04611898586153984 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880925416946411 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1587301641702652 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.022260624915361404 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5503899455070496 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16322927176952362 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.464782178401947 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.46189090609550476 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16843435168266296 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5622045993804932 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15738201141357422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5456712245941162 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5256702303886414 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03980669751763344 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_2_of_4‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14843828976154327 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5797064304351807 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_6‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14811520278453827 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655594 jet/accuracy_3_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49817588925361633 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4197562038898468 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29091373085975647 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908857345581055 particle/accuracy_2_of_2‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5263015627861023 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5924574732780457 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03951149433851242 particle/accuracy_2_of_3‚1 ,particle/sensitivity‚0.33898234367370605 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884761571884155 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9832519292831421 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +particle/specificity‚0.9520697593688965 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3348765969276428 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10386904329061508 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5990975499153137 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12899291515350342 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3324074149131775 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1949761062860489 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.568965494632721 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567901492118835 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22580084204673767 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35922619700431824 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6339285969734192 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4247775971889496 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 (particle/f_score‚0.42407962679862976 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2642804682254791 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5922566652297974 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5512802600860596 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964095950126648 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9866445064544678 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975565671920776 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.43589746952056885 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5399146676063538 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18501602113246918 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9530081748962402 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18843233585357666 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6163052320480347 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 $ _timestamp‚1.788457351436096e+09 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19276337325572968 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5509949326515198 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03479752317070961 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14613953232765198 particle/accuracy_3_of_5‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2560602128505707 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9669539928436279 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5671296715736389 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46702510118484497 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006527612917125225 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.509147584438324 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6050140857696533 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017019400373101234 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9906768202781677 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5027843713760376 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10045550018548965 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27868157625198364 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3863636255264282 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4485805332660675 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11424409598112106 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5955325365066528 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18625105917453766 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32113736867904663 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 *jet/accuracy_1_of_3‚0.8857831358909607 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826á=wvR…+ e44iwgzpj@ ‹ãæÔÖÜ0`Trainer.fit` stopped: `max_epochs=10` reached. ñV仟†+ e44iwgzpjŒ ŒãæÔàéŲzEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:24 • 0:00:00 1.39it/s v_num: wgzpõå %‡+ e44iwgzpj ŒãæÔ ïز €L2ˆ+- _wandbruntime‚9449  _runtime‚9449v‘’‰+‚(’éI